skiplock 1.0.8 → 1.0.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +60 -36
- data/bin/skiplock +26 -1
- data/lib/active_job/queue_adapters/skiplock_adapter.rb +2 -2
- data/lib/generators/skiplock/install_generator.rb +1 -1
- data/lib/generators/skiplock/templates/migration.rb.erb +30 -5
- data/lib/skiplock/counter.rb +4 -0
- data/lib/skiplock/cron.rb +3 -2
- data/lib/skiplock/dispatcher.rb +46 -76
- data/lib/skiplock/extension.rb +25 -0
- data/lib/skiplock/job.rb +72 -52
- data/lib/skiplock/manager.rb +121 -111
- data/lib/skiplock/version.rb +1 -1
- data/lib/skiplock/worker.rb +1 -1
- data/lib/skiplock.rb +26 -15
- metadata +5 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1acb8564f7930a8acbd316ae7ab85064aa3edb29232802d495f6c8261a2dfb72
|
4
|
+
data.tar.gz: c7de58da66f483450fb34680b5e83d37865ce02b9f7e500de386a2ce30c17edb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 69f54b48dd37b8814ef1bfb046945f4aaf13c996f35f9b297823f980804958d13a897dd1e8f5e28d6514d76867a4bac3a401aee2e682334cbabc2b52d4a36184
|
7
|
+
data.tar.gz: f2fda19cff5a11f6d51c7d2223de5a1251bf6744b86364c82343dafe02807ad59f02b0ad6a4324002e49393f0f87de1356686cda347172e6c947e523fcfe8c92
|
data/README.md
CHANGED
@@ -48,68 +48,76 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
|
|
48
48
|
```
|
49
49
|
2. `Skiplock` configuration
|
50
50
|
```yaml
|
51
|
-
# config/skiplock.yml
|
51
|
+
# config/skiplock.yml (default settings)
|
52
52
|
---
|
53
|
-
logging: timestamp
|
54
53
|
min_threads: 1
|
55
54
|
max_threads: 5
|
56
55
|
max_retries: 20
|
57
|
-
|
56
|
+
logfile: log/skiplock.log
|
57
|
+
loglevel: info
|
58
|
+
notification: custom
|
59
|
+
extensions: false
|
58
60
|
purge_completion: true
|
59
61
|
queues:
|
60
62
|
default: 200
|
61
|
-
mailers:
|
63
|
+
mailers: 999
|
62
64
|
workers: 0
|
63
65
|
```
|
64
66
|
Available configuration options are:
|
65
|
-
- **logging** (*enumeration*): sets the logging capability to **true** or **false**; setting to **timestamp** will enable logging with timestamps. The log files are: log/skiplock.log and log/skiplock.error.log
|
66
67
|
- **min_threads** (*integer*): sets minimum number of threads staying idle
|
67
68
|
- **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
|
68
|
-
- **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired
|
69
|
-
- **
|
69
|
+
- **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired. See `Retry system` for more details
|
70
|
+
- **logfile** (*string*): path filename for skiplock logs; empty logfile will disable logging
|
71
|
+
- **loglevel** (*string*): sets logging level (`debug, info, warn, error, fatal, unknown`)
|
72
|
+
- **notification** (*string*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, custom`); using `auto` will detect library if available. See `Notification system` for more details
|
73
|
+
- **extensions** (*boolean*): enable or disable the class method extension. See `ClassMethod extension` for more details
|
70
74
|
- **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
|
71
|
-
- **queues** (*hash*): defines the set of queues with
|
75
|
+
- **queues** (*hash*): defines the set of queues with priorities; lower priority takes precedence
|
72
76
|
- **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
|
73
|
-
|
74
|
-
#### Async mode
|
75
|
-
When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads. If using multi-worker cluster mode web server like Puma, then it should be configured as below:
|
76
|
-
```ruby
|
77
|
-
# config/puma.rb
|
78
|
-
before_fork do
|
79
|
-
# ...
|
80
|
-
Skiplock::Manager.shutdown
|
81
|
-
end
|
82
77
|
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
78
|
+
#### **Async mode**
|
79
|
+
When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads. If using multi-worker cluster mode web server like Puma, then all the Puma workers will also be able to perform `Skiplock` jobs.
|
80
|
+
|
81
|
+
#### **Standalone mode**
|
82
|
+
`Skiplock` standalone mode can be launched by using the `skiplock` executable; command line options can be provided to override the `Skiplock` configuration file.
|
83
|
+
```
|
84
|
+
$ bundle exec skiplock -h
|
85
|
+
Usage: skiplock [options]
|
86
|
+
-e, --environment STRING Rails environment
|
87
|
+
-l, --logfile STRING Full path to logfile
|
88
|
+
-s, --graceful-shutdown NUM Number of seconds to wait for graceful shutdown
|
89
|
+
-r, --max-retries NUM Number of maxixum retries
|
90
|
+
-t, --max-threads NUM Number of maximum threads
|
91
|
+
-T, --min-threads NUM Number of minimum threads
|
92
|
+
-w, --workers NUM Number of workers
|
93
|
+
-h, --help Show this message
|
88
94
|
```
|
89
95
|
|
90
96
|
## Usage
|
91
|
-
|
92
|
-
-
|
97
|
+
Inside the Rails application:
|
98
|
+
- queue your job
|
93
99
|
```ruby
|
94
100
|
MyJob.perform_later
|
95
101
|
```
|
96
|
-
- Skiplock supports all ActiveJob features
|
102
|
+
- Skiplock supports all ActiveJob features
|
97
103
|
```ruby
|
98
104
|
MyJob.set(queue: 'my_queue', wait: 5.minutes, priority: 10).perform_later(1,2,3)
|
99
105
|
```
|
100
|
-
|
106
|
+
Outside the Rails application:
|
107
|
+
- queue the jobs by inserting the job records directly to the database table
|
101
108
|
```sql
|
102
109
|
INSERT INTO skiplock.jobs(job_class) VALUES ('MyJob');
|
103
110
|
```
|
104
|
-
-
|
111
|
+
- with scheduling, priority, queue and arguments
|
105
112
|
```sql
|
106
|
-
INSERT INTO skiplock.jobs(job_class,queue_name,priority,scheduled_at,data)
|
113
|
+
INSERT INTO skiplock.jobs(job_class, queue_name, priority, scheduled_at, data)
|
114
|
+
VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min', '{"arguments":[1,2,3]}');
|
107
115
|
```
|
108
|
-
##
|
109
|
-
*Why do queues
|
116
|
+
## Queue priority vs Job priority
|
117
|
+
*Why do queues use priorities when jobs already have priorities?*
|
110
118
|
- Jobs are only prioritized with other jobs from the same queue
|
111
119
|
- Queues, on the other hand, are prioritized with other queues
|
112
|
-
- Rails has built-in queues that
|
120
|
+
- Rails has built-in queues that dispatch jobs without priorities (eg. Mail Delivery will queue as **mailers** with no priority)
|
113
121
|
|
114
122
|
## Cron system
|
115
123
|
`Skiplock` provides the capability to setup cron jobs for running tasks periodically. It fully supports the cron syntax to specify the frequency of the jobs. To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
|
@@ -131,7 +139,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
|
|
131
139
|
- to remove the cron schedule from the job, simply comment out the constant definition or delete the line then re-deploy the application. At startup, the cron jobs that were undefined will be removed automatically
|
132
140
|
|
133
141
|
## Retry system
|
134
|
-
`Skiplock` fully supports ActiveJob built-in retry system. It also has its own retry system for fallback. To use ActiveJob retry system, define the
|
142
|
+
`Skiplock` fully supports ActiveJob built-in retry system. It also has its own retry system for fallback. To use ActiveJob retry system, define the `retry_on` block per ActiveJob's documentation.
|
135
143
|
- configures `MyJob` to retry at maximum 20 attempts on StandardError with fixed delay of 5 seconds
|
136
144
|
```ruby
|
137
145
|
class MyJob < ActiveJob::Base
|
@@ -147,21 +155,37 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
|
|
147
155
|
# ...
|
148
156
|
end
|
149
157
|
```
|
150
|
-
|
158
|
+
If the retry attempt limit configured in ActiveJob has been reached, then the control will be passed back to `skiplock` to be marked as an expired job.
|
151
159
|
|
152
|
-
If the
|
160
|
+
If the `retry_on` block is not defined, then the built-in retry system of `skiplock` will kick in automatically. The retrying schedule is using an exponential formula (5 + 2**attempt). The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired. The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
|
153
161
|
|
154
162
|
## Notification system
|
155
|
-
`Skiplock` can use existing exception notification library to notify errors and exceptions. It supports `airbrake`, `bugsnag`, and `exception_notification`.
|
163
|
+
`Skiplock` can use existing exception notification library to notify errors and exceptions. It supports `airbrake`, `bugsnag`, and `exception_notification`. Custom notification can also be called whenever an exception occurs; it can be configured in an initializer like below:
|
156
164
|
```ruby
|
157
165
|
# config/initializers/skiplock.rb
|
158
|
-
Skiplock.on_error
|
166
|
+
Skiplock.on_error do |ex, previous|
|
159
167
|
if ex.backtrace != previous.try(:backtrace)
|
160
168
|
# sends custom email on new exceptions only
|
161
169
|
# the same repeated exceptions will only be sent once to avoid SPAM
|
170
|
+
# NOTE: exceptions generated from Job executions will not provide 'previous' exceptions
|
162
171
|
end
|
163
172
|
end
|
173
|
+
# supports multiple 'on_error' event callbacks
|
164
174
|
```
|
175
|
+
## ClassMethod extension
|
176
|
+
`Skiplock` can add extension to allow all class methods to be performed as a background job; it is disabled in the default configuration. To enable, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`.
|
177
|
+
- Queue class method `generate_thumbnails` of class `Image` as background job to run as soon as possible
|
178
|
+
```ruby
|
179
|
+
Image.skiplock.generate_thumbnails(height: 100, ratio: true)
|
180
|
+
```
|
181
|
+
- Queue class method `cleanup` of class `Session` as background job on queue `maintenance` to run after 5 minutes
|
182
|
+
```ruby
|
183
|
+
Session.skiplock(wait: 5.minutes, queue: 'maintenance').cleanup
|
184
|
+
```
|
185
|
+
- Queue class method `charge` of class `Subscription` as background job to run tomorrow at noon
|
186
|
+
```ruby
|
187
|
+
Subscription.skiplock(wait_until: Date.tomorrow.noon).charge(amount: 100)
|
188
|
+
```
|
165
189
|
|
166
190
|
## Contributing
|
167
191
|
|
data/bin/skiplock
CHANGED
@@ -1,3 +1,28 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
|
+
require 'optparse'
|
3
|
+
options = {}
|
4
|
+
begin
|
5
|
+
op = OptionParser.new do |opts|
|
6
|
+
opts.banner = "Usage: #{File.basename($0)} [options]"
|
7
|
+
opts.on('-e', '--environment STRING', String, 'Rails environment')
|
8
|
+
opts.on('-l', '--logfile STRING', String, 'Full path to logfile')
|
9
|
+
opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
|
10
|
+
opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
|
11
|
+
opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')
|
12
|
+
opts.on('-T', '--min-threads NUM', Integer, 'Number of minimum threads')
|
13
|
+
opts.on('-w', '--workers NUM', Integer, 'Number of workers')
|
14
|
+
opts.on_tail('-h', '--help', 'Show this message') do
|
15
|
+
exit
|
16
|
+
end
|
17
|
+
end
|
18
|
+
op.parse!(into: options)
|
19
|
+
rescue Exception => e
|
20
|
+
puts "\n#{e.message}\n\n" unless e.is_a?(SystemExit)
|
21
|
+
puts op
|
22
|
+
exit
|
23
|
+
end
|
24
|
+
options.transform_keys! { |k| k.to_s.gsub('-', '_').to_sym }
|
25
|
+
env = options.delete(:environment)
|
26
|
+
ENV['RAILS_ENV'] = env if env
|
2
27
|
require File.expand_path("config/environment.rb")
|
3
|
-
Skiplock::Manager.
|
28
|
+
Skiplock::Manager.new(**options.merge(standalone: true))
|
@@ -2,11 +2,11 @@ module ActiveJob
|
|
2
2
|
module QueueAdapters
|
3
3
|
class SkiplockAdapter
|
4
4
|
def initialize
|
5
|
-
Skiplock::Manager.
|
5
|
+
Rails.application.config.after_initialize { Skiplock::Manager.new }
|
6
6
|
end
|
7
7
|
|
8
8
|
def enqueue(job)
|
9
|
-
|
9
|
+
Skiplock::Job.enqueue(job)
|
10
10
|
end
|
11
11
|
|
12
12
|
def enqueue_at(job, timestamp)
|
@@ -1,6 +1,14 @@
|
|
1
1
|
class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]" %>
|
2
2
|
def up
|
3
3
|
execute 'CREATE SCHEMA skiplock'
|
4
|
+
create_table 'skiplock.counters', id: :uuid do |t|
|
5
|
+
t.integer :completions, null: false, default: 0
|
6
|
+
t.integer :dispatches, null: false, default: 0
|
7
|
+
t.integer :expiries, null: false, default: 0
|
8
|
+
t.integer :failures, null: false, default: 0
|
9
|
+
t.integer :retries, null: false, default: 0
|
10
|
+
t.date :day, null: false, index: { unique: true }
|
11
|
+
end
|
4
12
|
create_table 'skiplock.jobs', id: :uuid do |t|
|
5
13
|
t.uuid :worker_id, index: true
|
6
14
|
t.string :job_class, null: false
|
@@ -19,10 +27,11 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
|
|
19
27
|
t.timestamps null: false, default: -> { 'now()' }
|
20
28
|
end
|
21
29
|
create_table 'skiplock.workers', id: :uuid do |t|
|
22
|
-
t.integer :pid, null: false
|
23
|
-
t.integer :
|
30
|
+
t.integer :pid, null: false
|
31
|
+
t.integer :sid, null: false
|
24
32
|
t.integer :capacity, null: false
|
25
33
|
t.string :hostname, null: false, index: true
|
34
|
+
t.boolean :master, null: false, default: false, index: true
|
26
35
|
t.jsonb :data
|
27
36
|
t.timestamps null: false, index: true, default: -> { 'now()' }
|
28
37
|
end
|
@@ -31,10 +40,25 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
|
|
31
40
|
DECLARE
|
32
41
|
record RECORD;
|
33
42
|
BEGIN
|
43
|
+
record = NEW;
|
34
44
|
IF (TG_OP = 'DELETE') THEN
|
35
45
|
record = OLD;
|
36
|
-
|
37
|
-
|
46
|
+
IF (record.finished_at IS NOT NULL OR record.expired_at IS NOT NULL) THEN
|
47
|
+
RETURN NULL;
|
48
|
+
END IF;
|
49
|
+
INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
|
50
|
+
ELSIF (record.running = TRUE) THEN
|
51
|
+
IF (record.executions IS NULL) THEN
|
52
|
+
INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
|
53
|
+
ELSE
|
54
|
+
INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
|
55
|
+
END IF;
|
56
|
+
ELSIF (record.finished_at IS NOT NULL) THEN
|
57
|
+
INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
|
58
|
+
ELSIF (record.expired_at IS NOT NULL) THEN
|
59
|
+
INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
|
60
|
+
ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
|
61
|
+
INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
|
38
62
|
END IF;
|
39
63
|
PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
|
40
64
|
RETURN NULL;
|
@@ -51,7 +75,7 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
|
|
51
75
|
ELSE
|
52
76
|
record = NEW;
|
53
77
|
END IF;
|
54
|
-
PERFORM pg_notify('skiplock::workers', CONCAT(TG_OP,',',record.id::TEXT,',',record.hostname,',',record.capacity,',',record.pid,',',record.
|
78
|
+
PERFORM pg_notify('skiplock::workers', CONCAT(TG_OP,',',record.id::TEXT,',',record.hostname,',',record.master::TEXT,',',record.capacity,',',record.pid,',',record.sid,',',CAST(EXTRACT(EPOCH FROM record.created_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.updated_at) AS FLOAT)::TEXT));
|
55
79
|
RETURN NULL;
|
56
80
|
END;
|
57
81
|
$$ LANGUAGE plpgsql;
|
@@ -62,6 +86,7 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
|
|
62
86
|
execute "CREATE INDEX jobs_retry_index ON skiplock.jobs(scheduled_at) WHERE running = FALSE AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL"
|
63
87
|
execute "CREATE INDEX jobs_cron_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE cron IS NOT NULL AND finished_at IS NULL"
|
64
88
|
execute "CREATE UNIQUE INDEX jobs_unique_cron_index ON skiplock.jobs (job_class) WHERE cron IS NOT NULL"
|
89
|
+
execute "CREATE UNIQUE INDEX workers_unique_master_index ON skiplock.workers(hostname) WHERE master = 't'"
|
65
90
|
end
|
66
91
|
|
67
92
|
def down
|
data/lib/skiplock/cron.rb
CHANGED
@@ -2,17 +2,18 @@ require 'cron_parser'
|
|
2
2
|
module Skiplock
|
3
3
|
class Cron
|
4
4
|
def self.setup
|
5
|
+
Rails.application.eager_load! if Rails.env.development?
|
5
6
|
cronjobs = []
|
6
7
|
ActiveJob::Base.descendants.each do |j|
|
7
8
|
next unless j.const_defined?('CRON')
|
8
9
|
cron = j.const_get('CRON')
|
9
|
-
job = Job.find_by('job_class = ? AND cron IS NOT NULL', j.name) || Job.new(job_class: j.name, cron: cron)
|
10
|
+
job = Job.find_by('job_class = ? AND cron IS NOT NULL', j.name) || Job.new(job_class: j.name, cron: cron, locale: I18n.locale, timezone: Time.zone.name)
|
10
11
|
time = self.next_schedule_at(cron)
|
11
12
|
if time
|
12
13
|
job.cron = cron
|
13
14
|
job.running = false
|
14
15
|
job.scheduled_at = Time.at(time)
|
15
|
-
job.save
|
16
|
+
job.save
|
16
17
|
cronjobs << j.name
|
17
18
|
end
|
18
19
|
end
|
data/lib/skiplock/dispatcher.rb
CHANGED
@@ -1,52 +1,27 @@
|
|
1
1
|
module Skiplock
|
2
2
|
class Dispatcher
|
3
|
-
def initialize(
|
4
|
-
@
|
5
|
-
@
|
6
|
-
@
|
7
|
-
|
8
|
-
@worker_pids = worker_pids + [ Process.pid ]
|
9
|
-
else
|
10
|
-
@worker_num = worker_num
|
11
|
-
end
|
3
|
+
def initialize(worker:, worker_num: nil, **config)
|
4
|
+
@config = config
|
5
|
+
@worker = worker
|
6
|
+
@queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
|
7
|
+
@executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads], max_threads: @config[:max_threads], max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
|
12
8
|
@last_dispatch_at = 0
|
13
9
|
@next_schedule_at = Time.now.to_f
|
14
|
-
@
|
10
|
+
Process.setproctitle("skiplock-#{@worker.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
|
15
11
|
end
|
16
12
|
|
17
13
|
def run
|
14
|
+
@running = true
|
18
15
|
Thread.new do
|
19
|
-
sleep(1) while @running && !Rails.application.initialized?
|
20
|
-
Process.setproctitle("skiplock-#{@master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0
|
21
16
|
ActiveRecord::Base.connection_pool.with_connection do |connection|
|
22
17
|
connection.exec_query('LISTEN "skiplock::jobs"')
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
connection.exec_query('LISTEN "skiplock::workers"')
|
27
|
-
if File.exists?('tmp/cache/skiplock')
|
28
|
-
# get performed jobs that could not sync with database
|
29
|
-
job_ids = File.read('tmp/cache/skiplock').split("\n")
|
30
|
-
if Settings['purge_completion']
|
31
|
-
Job.where(id: job_ids).delete_all
|
32
|
-
else
|
33
|
-
Job.where(id: job_ids).update_all(running: false, finished_at: File.mtime('tmp/cache/skiplock'), updated_at: Time.now)
|
34
|
-
end
|
35
|
-
File.delete('tmp/cache/skiplock')
|
36
|
-
end
|
37
|
-
# get dead worker ids
|
38
|
-
dead_worker_ids = Worker.where(hostname: hostname).where.not(pid: @worker_pids).ids
|
39
|
-
if dead_worker_ids.count > 0
|
40
|
-
# reset orphaned jobs of the dead worker ids for retry
|
41
|
-
Job.where(running: true).where(worker_id: dead_worker_ids).update_all(running: false, worker_id: nil)
|
42
|
-
# remove dead workers
|
43
|
-
Worker.where(id: dead_worker_ids).delete_all
|
44
|
-
end
|
45
|
-
# reset retries schedules on startup
|
46
|
-
Job.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
|
18
|
+
if @worker.master
|
19
|
+
Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
|
20
|
+
check_sync_errors
|
47
21
|
Cron.setup
|
48
22
|
end
|
49
23
|
error = false
|
24
|
+
timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
50
25
|
while @running
|
51
26
|
begin
|
52
27
|
if error
|
@@ -54,92 +29,87 @@ module Skiplock
|
|
54
29
|
connection.reconnect!
|
55
30
|
sleep(0.5)
|
56
31
|
connection.exec_query('LISTEN "skiplock::jobs"')
|
57
|
-
connection.exec_query('LISTEN "skiplock::workers"') if @master
|
58
32
|
@next_schedule_at = Time.now.to_f
|
59
33
|
end
|
34
|
+
check_sync_errors
|
60
35
|
error = false
|
61
36
|
end
|
62
|
-
|
63
|
-
completed_ids = Job::Errors.keys.map { |k| k if Job::Errors[k] }.compact
|
64
|
-
if Settings['purge_completion'] && completed_ids.count > 0
|
65
|
-
Job.where(id: completed_ids, running: true).delete_all
|
66
|
-
elsif completed_ids.count > 0
|
67
|
-
Job.where(id: completed_ids, running: true).update_all(running: false, finished_at: Time.now, updated_at: Time.now)
|
68
|
-
end
|
69
|
-
orphaned_ids = Job::Errors.keys.map { |k| k unless Job::Errors[k] }.compact
|
70
|
-
Job.where(id: orphaned_ids, running: true).update_all(running: false, worker_id: nil, scheduled_at: (Time.now + 10), updated_at: Time.now) if orphaned_ids.count > 0
|
71
|
-
Job::Errors.clear
|
72
|
-
end
|
73
|
-
notifications = { 'skiplock::jobs' => [], 'skiplock::workers' => [] }
|
37
|
+
job_notifications = []
|
74
38
|
connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
|
75
|
-
|
39
|
+
job_notifications << payload if payload
|
76
40
|
loop do
|
77
41
|
payload = connection.raw_connection.notifies
|
78
42
|
break unless @running && payload
|
79
|
-
|
43
|
+
job_notifications << payload[:extra]
|
80
44
|
end
|
81
|
-
|
45
|
+
job_notifications.each do |n|
|
82
46
|
op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
|
83
|
-
if @master
|
84
|
-
# TODO: report job status to action cable
|
85
|
-
end
|
86
47
|
next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0 || scheduled_at.to_f < @last_dispatch_at
|
87
|
-
if scheduled_at.to_f
|
48
|
+
if scheduled_at.to_f <= Time.now.to_f
|
88
49
|
@next_schedule_at = Time.now.to_f
|
89
50
|
elsif scheduled_at.to_f < @next_schedule_at
|
90
51
|
@next_schedule_at = scheduled_at.to_f
|
91
52
|
end
|
92
53
|
end
|
93
|
-
if @master
|
94
|
-
# TODO: report worker status to action cable
|
95
|
-
notifications['skiplock::workers'].each do |n|
|
96
|
-
end
|
97
|
-
end
|
98
54
|
end
|
99
55
|
if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
|
100
56
|
@executor.post { do_work }
|
101
57
|
end
|
58
|
+
if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
|
59
|
+
@worker.touch
|
60
|
+
timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
61
|
+
end
|
102
62
|
rescue Exception => ex
|
103
63
|
# most likely error with database connection
|
104
|
-
|
105
|
-
|
106
|
-
Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
|
64
|
+
Skiplock.logger.error(ex)
|
65
|
+
Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
|
107
66
|
error = true
|
108
|
-
t =
|
67
|
+
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
109
68
|
while @running
|
110
69
|
sleep(0.5)
|
111
|
-
break if
|
70
|
+
break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > 5
|
112
71
|
end
|
113
72
|
@last_exception = ex
|
114
73
|
end
|
115
74
|
sleep(0.2)
|
116
75
|
end
|
117
76
|
connection.exec_query('UNLISTEN *')
|
77
|
+
@executor.shutdown
|
78
|
+
@executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
|
118
79
|
end
|
119
80
|
end
|
120
81
|
end
|
121
82
|
|
122
|
-
def shutdown
|
83
|
+
def shutdown
|
123
84
|
@running = false
|
124
|
-
@executor.shutdown
|
125
|
-
@executor.wait_for_termination if wait
|
126
|
-
@worker.delete if @worker
|
127
85
|
end
|
128
86
|
|
129
87
|
private
|
130
88
|
|
89
|
+
def check_sync_errors
|
90
|
+
# get performed jobs that could not sync with database
|
91
|
+
Dir.glob('tmp/skiplock/*').each do |f|
|
92
|
+
job_from_db = Job.find_by(id: File.basename(f), running: true)
|
93
|
+
disposed = true
|
94
|
+
if job_from_db
|
95
|
+
job, ex = YAML.load_file(f) rescue nil
|
96
|
+
disposed = job.dispose(ex, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
|
97
|
+
end
|
98
|
+
File.delete(f) if disposed
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
131
102
|
def do_work
|
132
103
|
while @running
|
133
|
-
@last_dispatch_at = Time.now.to_f
|
134
|
-
result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id)
|
135
|
-
next if result.is_a?(Job)
|
104
|
+
@last_dispatch_at = Time.now.to_f - 1 # 1 second allowance for time drift
|
105
|
+
result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
|
106
|
+
next if result.is_a?(Job) && Time.now.to_f >= @next_schedule_at
|
136
107
|
@next_schedule_at = result if result.is_a?(Float)
|
137
108
|
break
|
138
109
|
end
|
139
110
|
rescue Exception => ex
|
140
|
-
|
141
|
-
|
142
|
-
Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
|
111
|
+
Skiplock.logger.error(ex)
|
112
|
+
Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
|
143
113
|
@last_exception = ex
|
144
114
|
end
|
145
115
|
end
|
@@ -0,0 +1,25 @@
|
|
1
|
+
module Skiplock
|
2
|
+
module Extension
|
3
|
+
class Proxy < BasicObject
|
4
|
+
def initialize(target, options = {})
|
5
|
+
@target = target
|
6
|
+
@options = options
|
7
|
+
end
|
8
|
+
|
9
|
+
def method_missing(name, *args)
|
10
|
+
ProxyJob.set(@options).perform_later(::YAML.dump([ @target, name, args ]))
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
class ProxyJob < ActiveJob::Base
|
15
|
+
def perform(yml)
|
16
|
+
target, method_name, args = ::YAML.load(yml)
|
17
|
+
target.__send__(method_name, *args)
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def skiplock(options = {})
|
22
|
+
Proxy.new(self, options)
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
data/lib/skiplock/job.rb
CHANGED
@@ -1,73 +1,44 @@
|
|
1
1
|
module Skiplock
|
2
2
|
class Job < ActiveRecord::Base
|
3
|
-
self.
|
4
|
-
Errors = Concurrent::Map.new
|
3
|
+
self.implicit_order_column = 'created_at'
|
5
4
|
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
self.connection.exec_query('END')
|
13
|
-
return (job ? job.scheduled_at.to_f : Float::INFINITY)
|
5
|
+
def self.dispatch(queues_order_query: nil, worker_id: nil, purge_completion: true, max_retries: 20)
|
6
|
+
job = nil
|
7
|
+
self.transaction do
|
8
|
+
job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
|
9
|
+
return (job ? job.scheduled_at.to_f : Float::INFINITY) if job.nil? || job.scheduled_at.to_f > Time.now.to_f
|
10
|
+
job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
|
14
11
|
end
|
15
|
-
job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)} WHERE id = '#{job.id}' RETURNING *").first
|
16
|
-
self.connection.exec_query('END')
|
17
12
|
job.data ||= {}
|
18
13
|
job.exception_executions ||= {}
|
19
14
|
job_data = job.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => job.id, 'enqueued_at' => job.updated_at, 'arguments' => (job.data['arguments'] || []))
|
20
15
|
job.executions = (job.executions || 0) + 1
|
16
|
+
Skiplock.logger.info "[Skiplock] Performing #{job.job_class} (#{job.id}) from queue '#{job.queue_name || 'default'}'..."
|
21
17
|
Thread.current[:skiplock_dispatch_job] = job
|
18
|
+
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
22
19
|
begin
|
23
20
|
ActiveJob::Base.execute(job_data)
|
24
21
|
rescue Exception => ex
|
22
|
+
Skiplock.logger.error(ex)
|
25
23
|
end
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
job.save!
|
33
|
-
else
|
34
|
-
job.scheduled_at = Time.now + (5 * 2**job.executions)
|
35
|
-
job.save!
|
36
|
-
end
|
37
|
-
Skiplock.on_error.call(ex) if Skiplock.on_error.is_a?(Proc) && (job.executions % 3 == 1)
|
38
|
-
elsif job.exception_executions.key?('activejob_retry')
|
39
|
-
job.save!
|
40
|
-
elsif job.cron
|
41
|
-
job.data['last_cron_run'] = Time.now.utc.to_s
|
42
|
-
next_cron_at = Cron.next_schedule_at(job.cron)
|
43
|
-
if next_cron_at
|
44
|
-
job.executions = 1
|
45
|
-
job.exception_executions = nil
|
46
|
-
job.scheduled_at = Time.at(next_cron_at)
|
47
|
-
job.save!
|
48
|
-
else
|
49
|
-
job.delete
|
24
|
+
unless ex
|
25
|
+
end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
26
|
+
job_name = job.job_class
|
27
|
+
if job.job_class == 'Skiplock::Extension::ProxyJob'
|
28
|
+
target, method_name = ::YAML.load(job.data['arguments'].first)
|
29
|
+
job_name = "'#{target.name}.#{method_name}'"
|
50
30
|
end
|
51
|
-
|
52
|
-
job.delete
|
53
|
-
else
|
54
|
-
job.finished_at = Time.now
|
55
|
-
job.exception_executions = nil
|
56
|
-
job.save!
|
31
|
+
Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{job.id}) from queue '#{job.queue_name || 'default'}' in #{end_time - start_time} seconds"
|
57
32
|
end
|
58
|
-
job
|
59
|
-
rescue Exception => ex
|
60
|
-
if performed
|
61
|
-
Errors[job.id] = true
|
62
|
-
File.write('tmp/cache/skiplock', job.id + "\n", mode: 'a')
|
63
|
-
else
|
64
|
-
Errors[job.id] = false
|
65
|
-
end
|
66
|
-
raise ex
|
33
|
+
job.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
|
67
34
|
ensure
|
68
35
|
Thread.current[:skiplock_dispatch_job] = nil
|
69
36
|
end
|
70
37
|
|
38
|
+
def self.enqueue(activejob)
|
39
|
+
self.enqueue_at(activejob, nil)
|
40
|
+
end
|
41
|
+
|
71
42
|
def self.enqueue_at(activejob, timestamp)
|
72
43
|
timestamp = Time.at(timestamp) if timestamp
|
73
44
|
if Thread.current[:skiplock_dispatch_job].try(:id) == activejob.job_id
|
@@ -76,8 +47,57 @@ module Skiplock
|
|
76
47
|
Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
|
77
48
|
Thread.current[:skiplock_dispatch_job]
|
78
49
|
else
|
79
|
-
|
50
|
+
serialize = activejob.serialize
|
51
|
+
Job.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'] }, 'scheduled_at' => timestamp))
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
def self.reset_retry_schedules
|
56
|
+
self.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
|
57
|
+
end
|
58
|
+
|
59
|
+
def dispose(ex, purge_completion: true, max_retries: 20)
|
60
|
+
dup = self.dup
|
61
|
+
self.running = false
|
62
|
+
self.worker_id = nil
|
63
|
+
self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
|
64
|
+
if ex
|
65
|
+
self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
|
66
|
+
if self.executions >= max_retries || self.exception_executions.key?('activejob_retry')
|
67
|
+
self.expired_at = Time.now
|
68
|
+
else
|
69
|
+
self.scheduled_at = Time.now + (5 * 2**self.executions)
|
70
|
+
end
|
71
|
+
self.save!
|
72
|
+
Skiplock.on_errors.each { |p| p.call(ex) }
|
73
|
+
elsif self.exception_executions.try(:key?, 'activejob_retry')
|
74
|
+
self.save!
|
75
|
+
elsif self.cron
|
76
|
+
self.data ||= {}
|
77
|
+
self.data['crons'] = (self.data['crons'] || 0) + 1
|
78
|
+
self.data['last_cron_at'] = Time.now.utc.to_s
|
79
|
+
next_cron_at = Cron.next_schedule_at(self.cron)
|
80
|
+
if next_cron_at
|
81
|
+
self.executions = nil
|
82
|
+
self.exception_executions = nil
|
83
|
+
self.scheduled_at = Time.at(next_cron_at)
|
84
|
+
self.save!
|
85
|
+
else
|
86
|
+
Skiplock.logger.error "[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}"
|
87
|
+
self.delete
|
88
|
+
end
|
89
|
+
elsif purge_completion
|
90
|
+
self.delete
|
91
|
+
else
|
92
|
+
self.finished_at = Time.now
|
93
|
+
self.exception_executions = nil
|
94
|
+
self.save!
|
80
95
|
end
|
96
|
+
self
|
97
|
+
rescue Exception => e
|
98
|
+
Skiplock.logger.error(e)
|
99
|
+
File.write("tmp/skiplock/#{self.id}", [dup, ex].to_yaml)
|
100
|
+
nil
|
81
101
|
end
|
82
102
|
end
|
83
103
|
end
|
data/lib/skiplock/manager.rb
CHANGED
@@ -1,155 +1,165 @@
|
|
1
1
|
module Skiplock
|
2
2
|
class Manager
|
3
|
-
def
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
return unless standalone || restart || (caller.any?{|l| l =~ %r{/rack/}} && (Settings['workers'] == 0 || Rails.env.development?))
|
20
|
-
if standalone
|
21
|
-
self.standalone
|
3
|
+
def initialize(**config)
|
4
|
+
@config = Skiplock::DEFAULT_CONFIG.dup
|
5
|
+
@config.merge!(YAML.load_file('config/skiplock.yml')) rescue nil
|
6
|
+
@config.symbolize_keys!
|
7
|
+
@config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
|
8
|
+
@config.merge!(config)
|
9
|
+
Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
|
10
|
+
return unless @config[:standalone] || (caller.any?{ |l| l =~ %r{/rack/} } && (@config[:workers] == 0 || Rails.env.development?))
|
11
|
+
@config[:hostname] = `hostname -f`.strip
|
12
|
+
do_config
|
13
|
+
banner if @config[:standalone]
|
14
|
+
cleanup_workers
|
15
|
+
create_worker
|
16
|
+
ActiveJob::Base.logger = nil
|
17
|
+
if @config[:standalone]
|
18
|
+
standalone
|
22
19
|
else
|
23
|
-
|
24
|
-
|
25
|
-
at_exit
|
20
|
+
dispatcher = Dispatcher.new(worker: @worker, **@config)
|
21
|
+
thread = dispatcher.run
|
22
|
+
at_exit do
|
23
|
+
dispatcher.shutdown
|
24
|
+
thread.join(@config[:graceful_shutdown])
|
25
|
+
@worker.delete
|
26
|
+
end
|
26
27
|
end
|
27
|
-
|
28
|
+
rescue Exception => ex
|
29
|
+
@logger.error(ex)
|
30
|
+
end
|
31
|
+
|
32
|
+
private
|
33
|
+
|
34
|
+
def banner
|
35
|
+
title = "Skiplock #{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
|
36
|
+
@logger.info "-"*(title.length)
|
37
|
+
@logger.info title
|
38
|
+
@logger.info "-"*(title.length)
|
39
|
+
@logger.info "ClassMethod extensions: #{@config[:extensions]}"
|
40
|
+
@logger.info " Purge completion: #{@config[:purge_completion]}"
|
41
|
+
@logger.info " Notification: #{@config[:notification]}"
|
42
|
+
@logger.info " Max retries: #{@config[:max_retries]}"
|
43
|
+
@logger.info " Min threads: #{@config[:min_threads]}"
|
44
|
+
@logger.info " Max threads: #{@config[:max_threads]}"
|
45
|
+
@logger.info " Environment: #{Rails.env}"
|
46
|
+
@logger.info " Loglevel: #{@config[:loglevel]}"
|
47
|
+
@logger.info " Logfile: #{@config[:logfile] || '(disabled)'}"
|
48
|
+
@logger.info " Workers: #{@config[:workers]}"
|
49
|
+
@logger.info " Queues: #{@config[:queues].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if @config[:queues].is_a?(Hash)
|
50
|
+
@logger.info " PID: #{Process.pid}"
|
51
|
+
@logger.info "-"*(title.length)
|
52
|
+
@logger.warn "[Skiplock] Custom notification has no registered 'on_error' callback" if Skiplock.on_errors.count == 0
|
28
53
|
end
|
29
|
-
|
30
|
-
def
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
54
|
+
|
55
|
+
def cleanup_workers
|
56
|
+
delete_ids = []
|
57
|
+
Worker.where(hostname: @config[:hostname]).each do |worker|
|
58
|
+
sid = Process.getsid(worker.pid) rescue nil
|
59
|
+
delete_ids << worker.id if worker.sid != sid || worker.updated_at < 30.minutes.ago
|
60
|
+
end
|
61
|
+
if delete_ids.count > 0
|
62
|
+
Job.where(running: true, worker_id: delete_ids).update_all(running: false, worker_id: nil)
|
63
|
+
Worker.where(id: delete_ids).delete_all
|
36
64
|
end
|
37
65
|
end
|
38
66
|
|
39
|
-
|
67
|
+
def create_worker(pid: Process.pid, sid: Process.getsid(), master: true)
|
68
|
+
@worker = Worker.create!(pid: pid, sid: sid, master: master, hostname: @config[:hostname], capacity: @config[:max_threads])
|
69
|
+
rescue
|
70
|
+
@worker = Worker.create!(pid: pid, sid: sid, master: false, hostname: @config[:hostname], capacity: @config[:max_threads])
|
71
|
+
end
|
40
72
|
|
41
|
-
def
|
42
|
-
config =
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
73
|
+
def do_config
|
74
|
+
@config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
|
75
|
+
@config[:graceful_shutdown] = 300 if @config[:graceful_shutdown] > 300
|
76
|
+
@config[:graceful_shutdown] = nil if @config[:graceful_shutdown] < 0
|
77
|
+
@config[:max_retries] = 20 if @config[:max_retries] > 20
|
78
|
+
@config[:max_retries] = 0 if @config[:max_retries] < 0
|
79
|
+
@config[:max_threads] = 1 if @config[:max_threads] < 1
|
80
|
+
@config[:max_threads] = 20 if @config[:max_threads] > 20
|
81
|
+
@config[:min_threads] = 0 if @config[:min_threads] < 0
|
82
|
+
@config[:workers] = 0 if @config[:workers] < 0
|
83
|
+
@config[:workers] = 1 if @config[:standalone] && @config[:workers] <= 0
|
84
|
+
@logger = ActiveSupport::Logger.new(STDOUT)
|
85
|
+
@logger.level = @config[:loglevel].to_sym
|
86
|
+
Skiplock.logger = @logger
|
87
|
+
raise "Cannot create logfile '#{@config[:logfile]}'" if @config[:logfile] && !File.writable?(File.dirname(@config[:logfile]))
|
88
|
+
@config[:logfile] = nil if @config[:logfile].to_s.length == 0
|
89
|
+
if @config[:logfile]
|
90
|
+
@logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(@config[:logfile])))
|
91
|
+
if @config[:standalone]
|
92
|
+
Rails.logger.reopen('/dev/null')
|
93
|
+
Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
|
94
|
+
end
|
95
|
+
end
|
96
|
+
@config[:queues].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if @config[:queues].is_a?(Hash)
|
97
|
+
if @config[:notification] == 'auto'
|
47
98
|
if defined?(Airbrake)
|
48
|
-
|
99
|
+
@config[:notification] = 'airbrake'
|
49
100
|
elsif defined?(Bugsnag)
|
50
|
-
|
101
|
+
@config[:notification] = 'bugsnag'
|
51
102
|
elsif defined?(ExceptionNotifier)
|
52
|
-
|
103
|
+
@config[:notification] = 'exception_notification'
|
53
104
|
else
|
54
|
-
|
55
|
-
exit
|
105
|
+
raise "Unable to detect any known exception notification library. Please define custom 'on_error' event callbacks and change to 'custom' notification in 'config/skiplock.yml'"
|
56
106
|
end
|
107
|
+
end
|
108
|
+
case @config[:notification]
|
57
109
|
when 'airbrake'
|
58
110
|
raise 'airbrake gem not found' unless defined?(Airbrake)
|
59
|
-
Skiplock.on_error
|
111
|
+
Skiplock.on_error do |ex, previous|
|
112
|
+
Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace)
|
113
|
+
end
|
60
114
|
when 'bugsnag'
|
61
115
|
raise 'bugsnag gem not found' unless defined?(Bugsnag)
|
62
|
-
Skiplock.on_error
|
116
|
+
Skiplock.on_error do |ex, previous|
|
117
|
+
Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace)
|
118
|
+
end
|
63
119
|
when 'exception_notification'
|
64
120
|
raise 'exception_notification gem not found' unless defined?(ExceptionNotifier)
|
65
|
-
Skiplock.on_error
|
121
|
+
Skiplock.on_error do |ex, previous|
|
122
|
+
ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace)
|
123
|
+
end
|
124
|
+
else
|
125
|
+
@config[:notification] = 'custom'
|
66
126
|
end
|
67
|
-
|
68
|
-
STDERR.puts "Invalid configuration 'config/skiplock.yml': #{e.message}"
|
69
|
-
exit
|
127
|
+
Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
|
70
128
|
end
|
71
129
|
|
72
|
-
def
|
73
|
-
if Settings['logging']
|
74
|
-
log_timestamp = (Settings['logging'].to_s == 'timestamp')
|
75
|
-
logfile = File.open('log/skiplock.log', 'a')
|
76
|
-
logfile.sync = true
|
77
|
-
$stdout = Demux.new(logfile, STDOUT, timestamp: log_timestamp)
|
78
|
-
errfile = File.open('log/skiplock.error.log', 'a')
|
79
|
-
errfile.sync = true
|
80
|
-
$stderr = Demux.new(errfile, STDERR, timestamp: log_timestamp)
|
81
|
-
logger = ActiveSupport::Logger.new($stdout)
|
82
|
-
logger.level = Rails.logger.level
|
83
|
-
Rails.logger.reopen('/dev/null')
|
84
|
-
Rails.logger.extend(ActiveSupport::Logger.broadcast(logger))
|
85
|
-
end
|
86
|
-
title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
|
87
|
-
puts "-"*(title.length)
|
88
|
-
puts title
|
89
|
-
puts "-"*(title.length)
|
90
|
-
puts "Purge completion: #{Settings['purge_completion']}"
|
91
|
-
puts " Notification: #{Settings['notification']}"
|
92
|
-
puts " Max retries: #{Settings['max_retries']}"
|
93
|
-
puts " Min threads: #{Settings['min_threads']}"
|
94
|
-
puts " Max threads: #{Settings['max_threads']}"
|
95
|
-
puts " Environment: #{Rails.env}"
|
96
|
-
puts " Logging: #{Settings['logging']}"
|
97
|
-
puts " Workers: #{Settings['workers']}"
|
98
|
-
puts " Queues: #{Settings['queues'].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if Settings['queues'].is_a?(Hash)
|
99
|
-
puts " PID: #{Process.pid}"
|
100
|
-
puts "-"*(title.length)
|
130
|
+
def standalone
|
101
131
|
parent_id = Process.pid
|
102
132
|
shutdown = false
|
103
133
|
Signal.trap("INT") { shutdown = true }
|
104
134
|
Signal.trap("TERM") { shutdown = true }
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
135
|
+
(@config[:workers] - 1).times do |n|
|
136
|
+
fork do
|
137
|
+
sleep 1
|
138
|
+
worker = create_worker(master: false)
|
139
|
+
dispatcher = Dispatcher.new(worker: worker, worker_num: n + 1, **@config)
|
109
140
|
thread = dispatcher.run
|
110
141
|
loop do
|
111
142
|
sleep 0.5
|
112
143
|
break if shutdown || Process.ppid != parent_id
|
113
144
|
end
|
114
|
-
dispatcher.shutdown
|
115
|
-
thread.join
|
145
|
+
dispatcher.shutdown
|
146
|
+
thread.join(@config[:graceful_shutdown])
|
147
|
+
worker.delete
|
116
148
|
exit
|
117
149
|
end
|
118
150
|
end
|
119
|
-
|
120
|
-
dispatcher = Dispatcher.new(worker_pids: worker_pids)
|
151
|
+
dispatcher = Dispatcher.new(worker: @worker, **@config)
|
121
152
|
thread = dispatcher.run
|
122
153
|
loop do
|
123
154
|
sleep 0.5
|
124
155
|
break if shutdown
|
125
156
|
end
|
157
|
+
@logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
|
126
158
|
Process.waitall
|
127
|
-
dispatcher.shutdown
|
128
|
-
thread.join
|
129
|
-
|
130
|
-
|
131
|
-
class Demux
|
132
|
-
def initialize(*targets, timestamp: true)
|
133
|
-
@targets = targets
|
134
|
-
@timestamp = timestamp
|
135
|
-
end
|
136
|
-
|
137
|
-
def close
|
138
|
-
@targets.each(&:close)
|
139
|
-
end
|
140
|
-
|
141
|
-
def flush
|
142
|
-
@targets.each(&:flush)
|
143
|
-
end
|
144
|
-
|
145
|
-
def tty?
|
146
|
-
true
|
147
|
-
end
|
148
|
-
|
149
|
-
def write(*args)
|
150
|
-
args.prepend("[#{Time.now.utc}]: ") if @timestamp
|
151
|
-
@targets.each {|t| t.write(*args)}
|
152
|
-
end
|
159
|
+
dispatcher.shutdown
|
160
|
+
thread.join(@config[:graceful_shutdown])
|
161
|
+
@worker.delete
|
162
|
+
@logger.info "[Skiplock] Shutdown completed."
|
153
163
|
end
|
154
164
|
end
|
155
165
|
end
|
data/lib/skiplock/version.rb
CHANGED
data/lib/skiplock/worker.rb
CHANGED
data/lib/skiplock.rb
CHANGED
@@ -1,26 +1,37 @@
|
|
1
1
|
require 'active_job'
|
2
2
|
require 'active_job/queue_adapters/skiplock_adapter'
|
3
3
|
require 'active_record'
|
4
|
+
require 'skiplock/counter'
|
4
5
|
require 'skiplock/cron'
|
5
6
|
require 'skiplock/dispatcher'
|
6
|
-
require 'skiplock/
|
7
|
+
require 'skiplock/extension'
|
7
8
|
require 'skiplock/job'
|
9
|
+
require 'skiplock/manager'
|
8
10
|
require 'skiplock/worker'
|
9
11
|
require 'skiplock/version'
|
10
12
|
|
11
13
|
module Skiplock
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
14
|
+
DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'log/skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
|
15
|
+
|
16
|
+
def self.logger=(l)
|
17
|
+
@logger = l
|
18
|
+
end
|
19
|
+
|
20
|
+
def self.logger
|
21
|
+
@logger
|
22
|
+
end
|
23
|
+
|
24
|
+
def self.on_error(&block)
|
25
|
+
@on_errors ||= []
|
26
|
+
@on_errors << block
|
27
|
+
block
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.on_errors
|
31
|
+
@on_errors || []
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.table_name_prefix
|
35
|
+
'skiplock.'
|
36
|
+
end
|
26
37
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: skiplock
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.
|
4
|
+
version: 1.0.12
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Tin Vo
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-08-30 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -82,8 +82,10 @@ files:
|
|
82
82
|
- lib/generators/skiplock/install_generator.rb
|
83
83
|
- lib/generators/skiplock/templates/migration.rb.erb
|
84
84
|
- lib/skiplock.rb
|
85
|
+
- lib/skiplock/counter.rb
|
85
86
|
- lib/skiplock/cron.rb
|
86
87
|
- lib/skiplock/dispatcher.rb
|
88
|
+
- lib/skiplock/extension.rb
|
87
89
|
- lib/skiplock/job.rb
|
88
90
|
- lib/skiplock/manager.rb
|
89
91
|
- lib/skiplock/version.rb
|
@@ -107,7 +109,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
107
109
|
- !ruby/object:Gem::Version
|
108
110
|
version: '0'
|
109
111
|
requirements: []
|
110
|
-
rubygems_version: 3.
|
112
|
+
rubygems_version: 3.0.3
|
111
113
|
signing_key:
|
112
114
|
specification_version: 4
|
113
115
|
summary: ActiveJob Queue Adapter for PostgreSQL
|