skiplock 1.0.8 → 1.0.9
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +22 -21
- data/bin/skiplock +24 -1
- data/lib/active_job/queue_adapters/skiplock_adapter.rb +1 -1
- data/lib/generators/skiplock/templates/migration.rb.erb +25 -2
- data/lib/skiplock.rb +11 -2
- data/lib/skiplock/counter.rb +4 -0
- data/lib/skiplock/dispatcher.rb +93 -94
- data/lib/skiplock/job.rb +48 -48
- data/lib/skiplock/manager.rb +20 -11
- data/lib/skiplock/version.rb +1 -1
- data/lib/skiplock/worker.rb +0 -1
- metadata +4 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 7a975a3bb22318c6203edd59428a93d6a4abfa5ea1382f35abd774760467dc23
|
4
|
+
data.tar.gz: 79b787d23af0db0188aebf1b0361649d906d5f0cca4488d7cfcb3f61927d4781
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 407469700b1fab6c4ea702b83df0571fd46322ad32c8c8fe8ffc0269e5e93cd9f774b8bd4658021320732ca4de6d7609a64430cbc26bddc14c9f14cf184f92e3
|
7
|
+
data.tar.gz: 69a5fe7f5c88f21585a4b581d34f44183cb12f25bc0df6e7982497fb63c088316a251cc55c4e712b3cad799ec9aa758c8045d911fbeda393267ed39bc694e9a5
|
data/README.md
CHANGED
@@ -68,48 +68,48 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
|
|
68
68
|
- **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired (see Retry System for more details)
|
69
69
|
- **notification** (*enumeration*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, false`)
|
70
70
|
- **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
|
71
|
-
- **queues** (*hash*): defines the set of queues with
|
71
|
+
- **queues** (*hash*): defines the set of queues with priorities; lower priority takes precedence
|
72
72
|
- **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
|
73
73
|
|
74
74
|
#### Async mode
|
75
75
|
When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads. If using multi-worker cluster mode web server like Puma, then it should be configured as below:
|
76
76
|
```ruby
|
77
77
|
# config/puma.rb
|
78
|
-
|
79
|
-
|
80
|
-
Skiplock::Manager.shutdown
|
78
|
+
# ...
|
79
|
+
on_worker_fork do |worker_index|
|
80
|
+
Skiplock::Manager.shutdown if worker_index == 1
|
81
81
|
end
|
82
82
|
|
83
83
|
after_worker_fork do |worker_index|
|
84
84
|
# restarts skiplock after all Puma workers have been started
|
85
|
-
|
86
|
-
Skiplock::Manager.start if (worker_index + 1) == @options[:workers]
|
85
|
+
Skiplock::Manager.start(restart: true) if defined?(Skiplock) && worker_index + 1 == @options[:workers]
|
87
86
|
end
|
88
87
|
```
|
89
|
-
|
90
88
|
## Usage
|
91
|
-
|
92
|
-
-
|
89
|
+
Inside the Rails application:
|
90
|
+
- queue your job
|
93
91
|
```ruby
|
94
92
|
MyJob.perform_later
|
95
93
|
```
|
96
|
-
- Skiplock supports all ActiveJob features
|
94
|
+
- Skiplock supports all ActiveJob features
|
97
95
|
```ruby
|
98
96
|
MyJob.set(queue: 'my_queue', wait: 5.minutes, priority: 10).perform_later(1,2,3)
|
99
97
|
```
|
100
|
-
|
98
|
+
Outside the Rails application:
|
99
|
+
- queue the jobs by inserting the job records directly to the database table
|
101
100
|
```sql
|
102
101
|
INSERT INTO skiplock.jobs(job_class) VALUES ('MyJob');
|
103
102
|
```
|
104
|
-
-
|
103
|
+
- with scheduling, priority, queue and arguments
|
105
104
|
```sql
|
106
|
-
INSERT INTO skiplock.jobs(job_class,queue_name,priority,scheduled_at,data)
|
105
|
+
INSERT INTO skiplock.jobs(job_class, queue_name, priority, scheduled_at, data)
|
106
|
+
VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min', '{"arguments":[1,2,3]}');
|
107
107
|
```
|
108
|
-
##
|
109
|
-
*Why do queues
|
108
|
+
## Queue priority vs Job priority
|
109
|
+
*Why do queues use priorities when jobs already have priorities?*
|
110
110
|
- Jobs are only prioritized with other jobs from the same queue
|
111
111
|
- Queues, on the other hand, are prioritized with other queues
|
112
|
-
- Rails has built-in queues that
|
112
|
+
- Rails has built-in queues that dispatch jobs without priorities (eg. Mail Delivery will queue as **mailers** with no priority)
|
113
113
|
|
114
114
|
## Cron system
|
115
115
|
`Skiplock` provides the capability to setup cron jobs for running tasks periodically. It fully supports the cron syntax to specify the frequency of the jobs. To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
|
@@ -131,7 +131,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
|
|
131
131
|
- to remove the cron schedule from the job, simply comment out the constant definition or delete the line then re-deploy the application. At startup, the cron jobs that were undefined will be removed automatically
|
132
132
|
|
133
133
|
## Retry system
|
134
|
-
`Skiplock` fully supports ActiveJob built-in retry system. It also has its own retry system for fallback. To use ActiveJob retry system, define the
|
134
|
+
`Skiplock` fully supports ActiveJob built-in retry system. It also has its own retry system for fallback. To use ActiveJob retry system, define the `retry_on` block per ActiveJob's documentation.
|
135
135
|
- configures `MyJob` to retry at maximum 20 attempts on StandardError with fixed delay of 5 seconds
|
136
136
|
```ruby
|
137
137
|
class MyJob < ActiveJob::Base
|
@@ -147,20 +147,21 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
|
|
147
147
|
# ...
|
148
148
|
end
|
149
149
|
```
|
150
|
-
|
150
|
+
If the retry attempt limit configured in ActiveJob has been reached, then the control will be passed back to `skiplock` to be marked as an expired job.
|
151
151
|
|
152
|
-
If the
|
152
|
+
If the `retry_on` block is not defined, then the built-in retry system of `skiplock` will kick in automatically. The retrying schedule is using an exponential formula (5 + 2**attempt). The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired. The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
|
153
153
|
|
154
154
|
## Notification system
|
155
|
-
`Skiplock` can use existing exception notification library to notify errors and exceptions. It supports `airbrake`, `bugsnag`, and `exception_notification
|
155
|
+
`Skiplock` can use existing exception notification library to notify errors and exceptions. It supports `airbrake`, `bugsnag`, and `exception_notification` as shown in the **Configuration** section above. Custom function can also be called whenever an exception occurs; it can be configured in an initializer like below:
|
156
156
|
```ruby
|
157
157
|
# config/initializers/skiplock.rb
|
158
|
-
Skiplock.on_error
|
158
|
+
Skiplock.on_error do |ex, previous|
|
159
159
|
if ex.backtrace != previous.try(:backtrace)
|
160
160
|
# sends custom email on new exceptions only
|
161
161
|
# the same repeated exceptions will only be sent once to avoid SPAM
|
162
162
|
end
|
163
163
|
end
|
164
|
+
# supports multiple on_error callbacks
|
164
165
|
```
|
165
166
|
|
166
167
|
## Contributing
|
data/bin/skiplock
CHANGED
@@ -1,3 +1,26 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
|
+
require 'optparse'
|
3
|
+
options = {}
|
4
|
+
begin
|
5
|
+
op = OptionParser.new do |opts|
|
6
|
+
opts.banner = "Usage: #{File.basename($0)} [options]"
|
7
|
+
opts.on('-e', '--environment STRING', String, 'Rails environment')
|
8
|
+
opts.on('-l', '--logging STRING', String, 'Possible values: true, false, timestamp')
|
9
|
+
opts.on('-r', '--max_retries NUM', Integer, 'Number of maxixum retries')
|
10
|
+
opts.on('-t', '--max_threads NUM', Integer, 'Number of maximum threads')
|
11
|
+
opts.on('-T', '--min_threads NUM', Integer, 'Number of minimum threads')
|
12
|
+
opts.on('-w', '--workers NUM', Integer, 'Number of workers')
|
13
|
+
opts.on_tail('-h', '--help', 'Show this message') do
|
14
|
+
exit
|
15
|
+
end
|
16
|
+
end
|
17
|
+
op.parse!(into: options)
|
18
|
+
rescue Exception => e
|
19
|
+
puts "\n#{e.message}\n\n" unless e.is_a?(SystemExit)
|
20
|
+
puts op
|
21
|
+
exit
|
22
|
+
end
|
23
|
+
env = options.delete(:environment)
|
24
|
+
ENV['RAILS_ENV'] = env if env
|
2
25
|
require File.expand_path("config/environment.rb")
|
3
|
-
Skiplock::Manager.start(standalone: true)
|
26
|
+
Skiplock::Manager.start(standalone: true, **options)
|
@@ -1,6 +1,14 @@
|
|
1
1
|
class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]" %>
|
2
2
|
def up
|
3
3
|
execute 'CREATE SCHEMA skiplock'
|
4
|
+
create_table 'skiplock.counters', id: :uuid do |t|
|
5
|
+
t.integer :completions, null: false, default: 0
|
6
|
+
t.integer :dispatches, null: false, default: 0
|
7
|
+
t.integer :expiries, null: false, default: 0
|
8
|
+
t.integer :failures, null: false, default: 0
|
9
|
+
t.integer :retries, null: false, default: 0
|
10
|
+
t.date :day, null: false, index: { unique: true }
|
11
|
+
end
|
4
12
|
create_table 'skiplock.jobs', id: :uuid do |t|
|
5
13
|
t.uuid :worker_id, index: true
|
6
14
|
t.string :job_class, null: false
|
@@ -31,10 +39,25 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
|
|
31
39
|
DECLARE
|
32
40
|
record RECORD;
|
33
41
|
BEGIN
|
42
|
+
record = NEW;
|
34
43
|
IF (TG_OP = 'DELETE') THEN
|
35
44
|
record = OLD;
|
36
|
-
|
37
|
-
|
45
|
+
IF (record.finished_at IS NOT NULL OR record.expired_at IS NOT NULL) THEN
|
46
|
+
RETURN NULL;
|
47
|
+
END IF;
|
48
|
+
INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
|
49
|
+
ELSIF (record.running = TRUE) THEN
|
50
|
+
IF (record.executions IS NULL) THEN
|
51
|
+
INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
|
52
|
+
ELSE
|
53
|
+
INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
|
54
|
+
END IF;
|
55
|
+
ELSIF (record.finished_at IS NOT NULL) THEN
|
56
|
+
INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
|
57
|
+
ELSIF (record.expired_at IS NOT NULL) THEN
|
58
|
+
INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
|
59
|
+
ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
|
60
|
+
INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
|
38
61
|
END IF;
|
39
62
|
PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
|
40
63
|
RETURN NULL;
|
data/lib/skiplock.rb
CHANGED
@@ -1,10 +1,11 @@
|
|
1
1
|
require 'active_job'
|
2
2
|
require 'active_job/queue_adapters/skiplock_adapter'
|
3
3
|
require 'active_record'
|
4
|
+
require 'skiplock/counter'
|
4
5
|
require 'skiplock/cron'
|
5
6
|
require 'skiplock/dispatcher'
|
6
|
-
require 'skiplock/manager'
|
7
7
|
require 'skiplock/job'
|
8
|
+
require 'skiplock/manager'
|
8
9
|
require 'skiplock/worker'
|
9
10
|
require 'skiplock/version'
|
10
11
|
|
@@ -22,5 +23,13 @@ module Skiplock
|
|
22
23
|
},
|
23
24
|
'workers' => 0
|
24
25
|
}
|
25
|
-
|
26
|
+
mattr_reader :on_errors, default: []
|
27
|
+
|
28
|
+
def self.on_error(&block)
|
29
|
+
@@on_errors << block
|
30
|
+
end
|
31
|
+
|
32
|
+
def self.table_name_prefix
|
33
|
+
'skiplock.'
|
34
|
+
end
|
26
35
|
end
|
data/lib/skiplock/dispatcher.rb
CHANGED
@@ -16,121 +16,120 @@ module Skiplock
|
|
16
16
|
|
17
17
|
def run
|
18
18
|
Thread.new do
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
if
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
19
|
+
Rails.application.reloader.wrap do
|
20
|
+
sleep(1) while @running && !Rails.application.initialized?
|
21
|
+
Rails.application.eager_load!
|
22
|
+
Process.setproctitle("skiplock-#{@master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0 && !Rails.env.development?
|
23
|
+
ActiveRecord::Base.connection_pool.with_connection do |connection|
|
24
|
+
connection.exec_query('LISTEN "skiplock::jobs"')
|
25
|
+
hostname = `hostname -f`.strip
|
26
|
+
@worker = Worker.create!(pid: Process.pid, ppid: (@master ? nil : Process.ppid), capacity: Settings['max_threads'], hostname: hostname)
|
27
|
+
if @master
|
28
|
+
connection.exec_query('LISTEN "skiplock::workers"')
|
29
|
+
Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
|
30
|
+
check_sync_errors
|
31
|
+
# get dead worker ids
|
32
|
+
dead_worker_ids = Worker.where(hostname: hostname).where.not(pid: @worker_pids).ids
|
33
|
+
if dead_worker_ids.count > 0
|
34
|
+
# reset orphaned jobs of the dead worker ids for retry
|
35
|
+
Job.where(running: true).where(worker_id: dead_worker_ids).update_all(running: false, worker_id: nil)
|
36
|
+
# remove dead workers
|
37
|
+
Worker.where(id: dead_worker_ids).delete_all
|
34
38
|
end
|
35
|
-
|
39
|
+
# reset retries schedules on startup
|
40
|
+
Job.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
|
41
|
+
Cron.setup
|
36
42
|
end
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
Job.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
|
47
|
-
Cron.setup
|
48
|
-
end
|
49
|
-
error = false
|
50
|
-
while @running
|
51
|
-
begin
|
52
|
-
if error
|
53
|
-
unless connection.active?
|
54
|
-
connection.reconnect!
|
55
|
-
sleep(0.5)
|
56
|
-
connection.exec_query('LISTEN "skiplock::jobs"')
|
57
|
-
connection.exec_query('LISTEN "skiplock::workers"') if @master
|
58
|
-
@next_schedule_at = Time.now.to_f
|
59
|
-
end
|
60
|
-
error = false
|
61
|
-
end
|
62
|
-
if Job::Errors.keys.count > 0
|
63
|
-
completed_ids = Job::Errors.keys.map { |k| k if Job::Errors[k] }.compact
|
64
|
-
if Settings['purge_completion'] && completed_ids.count > 0
|
65
|
-
Job.where(id: completed_ids, running: true).delete_all
|
66
|
-
elsif completed_ids.count > 0
|
67
|
-
Job.where(id: completed_ids, running: true).update_all(running: false, finished_at: Time.now, updated_at: Time.now)
|
68
|
-
end
|
69
|
-
orphaned_ids = Job::Errors.keys.map { |k| k unless Job::Errors[k] }.compact
|
70
|
-
Job.where(id: orphaned_ids, running: true).update_all(running: false, worker_id: nil, scheduled_at: (Time.now + 10), updated_at: Time.now) if orphaned_ids.count > 0
|
71
|
-
Job::Errors.clear
|
72
|
-
end
|
73
|
-
notifications = { 'skiplock::jobs' => [], 'skiplock::workers' => [] }
|
74
|
-
connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
|
75
|
-
notifications[channel] << payload if payload
|
76
|
-
loop do
|
77
|
-
payload = connection.raw_connection.notifies
|
78
|
-
break unless @running && payload
|
79
|
-
notifications[payload[:relname]] << payload[:extra]
|
80
|
-
end
|
81
|
-
notifications['skiplock::jobs'].each do |n|
|
82
|
-
op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
|
83
|
-
if @master
|
84
|
-
# TODO: report job status to action cable
|
85
|
-
end
|
86
|
-
next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0 || scheduled_at.to_f < @last_dispatch_at
|
87
|
-
if scheduled_at.to_f < Time.now.to_f
|
43
|
+
error = false
|
44
|
+
while @running
|
45
|
+
begin
|
46
|
+
if error
|
47
|
+
unless connection.active?
|
48
|
+
connection.reconnect!
|
49
|
+
sleep(0.5)
|
50
|
+
connection.exec_query('LISTEN "skiplock::jobs"')
|
51
|
+
connection.exec_query('LISTEN "skiplock::workers"') if @master
|
88
52
|
@next_schedule_at = Time.now.to_f
|
89
|
-
elsif scheduled_at.to_f < @next_schedule_at
|
90
|
-
@next_schedule_at = scheduled_at.to_f
|
91
53
|
end
|
54
|
+
check_sync_errors
|
55
|
+
error = false
|
92
56
|
end
|
93
|
-
|
94
|
-
|
95
|
-
notifications[
|
57
|
+
notifications = { 'skiplock::jobs' => [], 'skiplock::workers' => [] }
|
58
|
+
connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
|
59
|
+
notifications[channel] << payload if payload
|
60
|
+
loop do
|
61
|
+
payload = connection.raw_connection.notifies
|
62
|
+
break unless @running && payload
|
63
|
+
notifications[payload[:relname]] << payload[:extra]
|
64
|
+
end
|
65
|
+
notifications['skiplock::jobs'].each do |n|
|
66
|
+
op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
|
67
|
+
if @master
|
68
|
+
# TODO: report job status to action cable
|
69
|
+
end
|
70
|
+
next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0 || scheduled_at.to_f < @last_dispatch_at
|
71
|
+
if scheduled_at.to_f < Time.now.to_f
|
72
|
+
@next_schedule_at = Time.now.to_f
|
73
|
+
elsif scheduled_at.to_f < @next_schedule_at
|
74
|
+
@next_schedule_at = scheduled_at.to_f
|
75
|
+
end
|
76
|
+
end
|
77
|
+
if @master
|
78
|
+
# TODO: report worker status to action cable
|
79
|
+
notifications['skiplock::workers'].each do |n|
|
80
|
+
end
|
96
81
|
end
|
97
82
|
end
|
83
|
+
if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
|
84
|
+
@executor.post { do_work }
|
85
|
+
end
|
86
|
+
rescue Exception => ex
|
87
|
+
# most likely error with database connection
|
88
|
+
STDERR.puts ex.message
|
89
|
+
STDERR.puts ex.backtrace
|
90
|
+
Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
|
91
|
+
error = true
|
92
|
+
t = Time.now
|
93
|
+
while @running
|
94
|
+
sleep(0.5)
|
95
|
+
break if Time.now - t > 5
|
96
|
+
end
|
97
|
+
@last_exception = ex
|
98
98
|
end
|
99
|
-
|
100
|
-
@executor.post { do_work }
|
101
|
-
end
|
102
|
-
rescue Exception => ex
|
103
|
-
# most likely error with database connection
|
104
|
-
STDERR.puts ex.message
|
105
|
-
STDERR.puts ex.backtrace
|
106
|
-
Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
|
107
|
-
error = true
|
108
|
-
t = Time.now
|
109
|
-
while @running
|
110
|
-
sleep(0.5)
|
111
|
-
break if Time.now - t > 5
|
112
|
-
end
|
113
|
-
@last_exception = ex
|
99
|
+
sleep(0.2)
|
114
100
|
end
|
115
|
-
|
101
|
+
connection.exec_query('UNLISTEN *')
|
102
|
+
@executor.shutdown
|
103
|
+
@executor.wait_for_termination if @wait
|
104
|
+
@worker.delete if @worker
|
116
105
|
end
|
117
|
-
connection.exec_query('UNLISTEN *')
|
118
106
|
end
|
119
107
|
end
|
120
108
|
end
|
121
109
|
|
122
110
|
def shutdown(wait: true)
|
123
111
|
@running = false
|
124
|
-
@
|
125
|
-
@executor.wait_for_termination if wait
|
126
|
-
@worker.delete if @worker
|
112
|
+
@wait = wait
|
127
113
|
end
|
128
114
|
|
129
115
|
private
|
130
116
|
|
117
|
+
def check_sync_errors
|
118
|
+
# get performed jobs that could not sync with database
|
119
|
+
Dir.glob('tmp/skiplock/*').each do |f|
|
120
|
+
job_from_db = Job.find_by(id: File.basename(f), running: true)
|
121
|
+
disposed = true
|
122
|
+
if job_from_db
|
123
|
+
job, ex = YAML.load_file(f) rescue nil
|
124
|
+
disposed = job.dispose(ex)
|
125
|
+
end
|
126
|
+
File.delete(f) if disposed
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
131
130
|
def do_work
|
132
131
|
while @running
|
133
|
-
@last_dispatch_at = Time.now.to_f
|
132
|
+
@last_dispatch_at = Time.now.to_f - 1 # 1 second allowance for timedrift
|
134
133
|
result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id)
|
135
134
|
next if result.is_a?(Job)
|
136
135
|
@next_schedule_at = result if result.is_a?(Float)
|
@@ -139,7 +138,7 @@ module Skiplock
|
|
139
138
|
rescue Exception => ex
|
140
139
|
STDERR.puts ex.message
|
141
140
|
STDERR.puts ex.backtrace
|
142
|
-
Skiplock.
|
141
|
+
Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
|
143
142
|
@last_exception = ex
|
144
143
|
end
|
145
144
|
end
|
data/lib/skiplock/job.rb
CHANGED
@@ -1,18 +1,13 @@
|
|
1
1
|
module Skiplock
|
2
2
|
class Job < ActiveRecord::Base
|
3
|
-
self.table_name = 'skiplock.jobs'
|
4
|
-
Errors = Concurrent::Map.new
|
5
|
-
|
6
|
-
# Return: Skiplock::Job if it was executed; otherwise returns the next Job's schedule time in FLOAT
|
7
3
|
def self.dispatch(queues_order_query: nil, worker_id: nil)
|
8
|
-
performed = false
|
9
4
|
self.connection.exec_query('BEGIN')
|
10
5
|
job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
|
11
6
|
if job.nil? || job.scheduled_at.to_f > Time.now.to_f
|
12
7
|
self.connection.exec_query('END')
|
13
8
|
return (job ? job.scheduled_at.to_f : Float::INFINITY)
|
14
9
|
end
|
15
|
-
job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)} WHERE id = '#{job.id}' RETURNING *").first
|
10
|
+
job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
|
16
11
|
self.connection.exec_query('END')
|
17
12
|
job.data ||= {}
|
18
13
|
job.exception_executions ||= {}
|
@@ -23,51 +18,15 @@ module Skiplock
|
|
23
18
|
ActiveJob::Base.execute(job_data)
|
24
19
|
rescue Exception => ex
|
25
20
|
end
|
26
|
-
|
27
|
-
job.running = false
|
28
|
-
if ex
|
29
|
-
job.exception_executions["[#{ex.class.name}]"] = (job.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless job.exception_executions.key?('activejob_retry')
|
30
|
-
if job.executions >= Settings['max_retries'] || job.exception_executions.key?('activejob_retry')
|
31
|
-
job.expired_at = Time.now
|
32
|
-
job.save!
|
33
|
-
else
|
34
|
-
job.scheduled_at = Time.now + (5 * 2**job.executions)
|
35
|
-
job.save!
|
36
|
-
end
|
37
|
-
Skiplock.on_error.call(ex) if Skiplock.on_error.is_a?(Proc) && (job.executions % 3 == 1)
|
38
|
-
elsif job.exception_executions.key?('activejob_retry')
|
39
|
-
job.save!
|
40
|
-
elsif job.cron
|
41
|
-
job.data['last_cron_run'] = Time.now.utc.to_s
|
42
|
-
next_cron_at = Cron.next_schedule_at(job.cron)
|
43
|
-
if next_cron_at
|
44
|
-
job.executions = 1
|
45
|
-
job.exception_executions = nil
|
46
|
-
job.scheduled_at = Time.at(next_cron_at)
|
47
|
-
job.save!
|
48
|
-
else
|
49
|
-
job.delete
|
50
|
-
end
|
51
|
-
elsif Settings['purge_completion']
|
52
|
-
job.delete
|
53
|
-
else
|
54
|
-
job.finished_at = Time.now
|
55
|
-
job.exception_executions = nil
|
56
|
-
job.save!
|
57
|
-
end
|
58
|
-
job
|
59
|
-
rescue Exception => ex
|
60
|
-
if performed
|
61
|
-
Errors[job.id] = true
|
62
|
-
File.write('tmp/cache/skiplock', job.id + "\n", mode: 'a')
|
63
|
-
else
|
64
|
-
Errors[job.id] = false
|
65
|
-
end
|
66
|
-
raise ex
|
21
|
+
job.dispose(ex)
|
67
22
|
ensure
|
68
23
|
Thread.current[:skiplock_dispatch_job] = nil
|
69
24
|
end
|
70
25
|
|
26
|
+
def self.enqueue(activejob)
|
27
|
+
self.enqueue_at(activejob, nil)
|
28
|
+
end
|
29
|
+
|
71
30
|
def self.enqueue_at(activejob, timestamp)
|
72
31
|
timestamp = Time.at(timestamp) if timestamp
|
73
32
|
if Thread.current[:skiplock_dispatch_job].try(:id) == activejob.job_id
|
@@ -76,8 +35,49 @@ module Skiplock
|
|
76
35
|
Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
|
77
36
|
Thread.current[:skiplock_dispatch_job]
|
78
37
|
else
|
79
|
-
Job.create!(id: activejob.job_id, job_class: activejob.class.name, queue_name: activejob.queue_name, locale: activejob.locale, timezone: activejob.timezone, priority: activejob.priority,
|
38
|
+
Job.create!(id: activejob.job_id, job_class: activejob.class.name, queue_name: activejob.queue_name, locale: activejob.locale, timezone: activejob.timezone, priority: activejob.priority, data: { 'arguments' => activejob.serialize['arguments'] }, scheduled_at: timestamp)
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def dispose(ex)
|
43
|
+
self.running = false
|
44
|
+
self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
|
45
|
+
if ex
|
46
|
+
self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
|
47
|
+
if self.executions >= Settings['max_retries'] || self.exception_executions.key?('activejob_retry')
|
48
|
+
self.expired_at = Time.now
|
49
|
+
self.save!
|
50
|
+
else
|
51
|
+
self.scheduled_at = Time.now + (5 * 2**self.executions)
|
52
|
+
self.save!
|
53
|
+
end
|
54
|
+
Skiplock.on_errors.each { |p| p.call(ex) }
|
55
|
+
elsif self.exception_executions.try(:key?, 'activejob_retry')
|
56
|
+
self.save!
|
57
|
+
elsif self.cron
|
58
|
+
self.data ||= {}
|
59
|
+
self.data['crons'] = (self.data['crons'] || 0) + 1
|
60
|
+
self.data['last_cron_at'] = Time.now.utc.to_s
|
61
|
+
next_cron_at = Cron.next_schedule_at(self.cron)
|
62
|
+
if next_cron_at
|
63
|
+
self.executions = nil
|
64
|
+
self.exception_executions = nil
|
65
|
+
self.scheduled_at = Time.at(next_cron_at)
|
66
|
+
self.save!
|
67
|
+
else
|
68
|
+
self.delete
|
69
|
+
end
|
70
|
+
elsif Settings['purge_completion']
|
71
|
+
self.delete
|
72
|
+
else
|
73
|
+
self.finished_at = Time.now
|
74
|
+
self.exception_executions = nil
|
75
|
+
self.save!
|
80
76
|
end
|
77
|
+
self
|
78
|
+
rescue
|
79
|
+
File.write("tmp/skiplock/#{self.id}", [self, ex].to_yaml)
|
80
|
+
nil
|
81
81
|
end
|
82
82
|
end
|
83
83
|
end
|
data/lib/skiplock/manager.rb
CHANGED
@@ -14,9 +14,10 @@ module Skiplock
|
|
14
14
|
Settings['max_threads'] = 20 if Settings['max_threads'] > 20
|
15
15
|
Settings['min_threads'] = 0 if Settings['min_threads'] < 0
|
16
16
|
Settings['workers'] = 0 if Settings['workers'] < 0
|
17
|
+
Settings['workers'] = 1 if standalone && Settings['workers'] <= 0
|
17
18
|
Settings.freeze
|
18
19
|
end
|
19
|
-
return unless standalone || restart || (caller.any?{|l| l =~ %r{/rack/}} &&
|
20
|
+
return unless standalone || restart || (caller.any?{|l| l =~ %r{/rack/}} && Settings['workers'] == 0)
|
20
21
|
if standalone
|
21
22
|
self.standalone
|
22
23
|
else
|
@@ -42,27 +43,35 @@ module Skiplock
|
|
42
43
|
config = YAML.load_file('config/skiplock.yml') rescue {}
|
43
44
|
Settings.merge!(config)
|
44
45
|
Settings['queues'].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if Settings['queues'].is_a?(Hash)
|
45
|
-
|
46
|
-
|
46
|
+
@notification = Settings['notification'] = Settings['notification'].to_s.downcase
|
47
|
+
if @notification == 'auto'
|
47
48
|
if defined?(Airbrake)
|
48
|
-
|
49
|
+
@notification = 'airbrake'
|
49
50
|
elsif defined?(Bugsnag)
|
50
|
-
|
51
|
+
@notification = 'bugsnag'
|
51
52
|
elsif defined?(ExceptionNotifier)
|
52
|
-
|
53
|
+
@notification = 'exception_notification'
|
53
54
|
else
|
54
|
-
puts "Unable to detect any known exception notification gem. Please define custom 'on_error' function and disable notification in 'config/skiplock.yml'"
|
55
|
+
puts "Unable to detect any known exception notification gem. Please define custom 'on_error' callback function and disable 'auto' notification in 'config/skiplock.yml'"
|
55
56
|
exit
|
56
57
|
end
|
58
|
+
end
|
59
|
+
case @notification
|
57
60
|
when 'airbrake'
|
58
61
|
raise 'airbrake gem not found' unless defined?(Airbrake)
|
59
|
-
Skiplock.on_error
|
62
|
+
Skiplock.on_error do |ex, previous|
|
63
|
+
Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace)
|
64
|
+
end
|
60
65
|
when 'bugsnag'
|
61
66
|
raise 'bugsnag gem not found' unless defined?(Bugsnag)
|
62
|
-
Skiplock.on_error
|
67
|
+
Skiplock.on_error do |ex, previous|
|
68
|
+
Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace)
|
69
|
+
end
|
63
70
|
when 'exception_notification'
|
64
71
|
raise 'exception_notification gem not found' unless defined?(ExceptionNotifier)
|
65
|
-
Skiplock.on_error
|
72
|
+
Skiplock.on_error do |ex, previous|
|
73
|
+
ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace)
|
74
|
+
end
|
66
75
|
end
|
67
76
|
rescue Exception => e
|
68
77
|
STDERR.puts "Invalid configuration 'config/skiplock.yml': #{e.message}"
|
@@ -88,7 +97,7 @@ module Skiplock
|
|
88
97
|
puts title
|
89
98
|
puts "-"*(title.length)
|
90
99
|
puts "Purge completion: #{Settings['purge_completion']}"
|
91
|
-
puts " Notification: #{Settings['notification']}"
|
100
|
+
puts " Notification: #{Settings['notification']}#{(' (' + @notification + ')') if Settings['notification'] == 'auto'}"
|
92
101
|
puts " Max retries: #{Settings['max_retries']}"
|
93
102
|
puts " Min threads: #{Settings['min_threads']}"
|
94
103
|
puts " Max threads: #{Settings['max_threads']}"
|
data/lib/skiplock/version.rb
CHANGED
data/lib/skiplock/worker.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: skiplock
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.
|
4
|
+
version: 1.0.9
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Tin Vo
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-05-08 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -82,6 +82,7 @@ files:
|
|
82
82
|
- lib/generators/skiplock/install_generator.rb
|
83
83
|
- lib/generators/skiplock/templates/migration.rb.erb
|
84
84
|
- lib/skiplock.rb
|
85
|
+
- lib/skiplock/counter.rb
|
85
86
|
- lib/skiplock/cron.rb
|
86
87
|
- lib/skiplock/dispatcher.rb
|
87
88
|
- lib/skiplock/job.rb
|
@@ -107,7 +108,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
107
108
|
- !ruby/object:Gem::Version
|
108
109
|
version: '0'
|
109
110
|
requirements: []
|
110
|
-
rubygems_version: 3.
|
111
|
+
rubygems_version: 3.0.3
|
111
112
|
signing_key:
|
112
113
|
specification_version: 4
|
113
114
|
summary: ActiveJob Queue Adapter for PostgreSQL
|