workhorse 0.6.9 → 1.0.0.beta0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +34 -0
- data/FAQ.md +5 -1
- data/README.md +40 -2
- data/Rakefile +1 -0
- data/VERSION +1 -1
- data/lib/workhorse.rb +1 -0
- data/lib/workhorse/db_job.rb +7 -0
- data/lib/workhorse/jobs/cleanup_succeeded_jobs.rb +1 -1
- data/lib/workhorse/jobs/detect_stale_jobs_job.rb +48 -0
- data/lib/workhorse/poller.rb +64 -46
- data/test/lib/test_helper.rb +10 -1
- data/test/workhorse/poller_test.rb +79 -0
- data/workhorse.gemspec +8 -5
- metadata +19 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5dd13b11d5f7488e58812f57ad2bb70da07b95f18b89653687bf9ec579bbcaa5
|
4
|
+
data.tar.gz: a0e7c53cf4de56b6109830decf16f6f70a8aead13ca703a795ce7709234c1b05
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2c6e9162d5b2735cd1de678b56e4c4e4960e25776a64cc45dfe5fbfee4fe0e50b1b3a5d15e81ca65ebfe496cce6ac4d81cfe1ba47a65b591a54916807bbe5c68
|
7
|
+
data.tar.gz: a24094b51e7751010920e39ad6ac3924dbccdfb84f1728efec73ed1cc0e9376a873cdb2ff4a562f49d78b17ae08e5659fd596684b299fc35b8bdcd228be15f0e
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,39 @@
|
|
1
1
|
# Workhorse Changelog
|
2
2
|
|
3
|
+
## 1.0.0.beta0 - 2020-08-19
|
4
|
+
|
5
|
+
This is a stability release that is still experimental and has to be tested in
|
6
|
+
battle before it can be considered stable.
|
7
|
+
|
8
|
+
* Simplify locking during polling. Other than locking individual jobs, pollers
|
9
|
+
now acquire a global lock. While this can lead to many pollers waiting for
|
10
|
+
each others locks, performing a poll is usually done very quickly and the
|
11
|
+
performance drawback is to be considered neglegible. This change should work
|
12
|
+
around some deadlock issues as well as an issue where a job was obtained by
|
13
|
+
more than one poller.
|
14
|
+
|
15
|
+
* Shut down worker if polling encountered any kind of error (running jobs will
|
16
|
+
be completed whenever possible). This leads to potential watcher jobs being
|
17
|
+
able to restore the failed process.
|
18
|
+
|
19
|
+
* Make unit test database connection configurable using environment variables
|
20
|
+
`DB_NAME`, `DB_USERNAME`, `DB_PASSWORD` and `DB_HOST`. This is only relevant
|
21
|
+
if you are working on workhorse and need to run the unit tests.
|
22
|
+
|
23
|
+
* Fix misbehaviour where queueless jobs were not picked up by workers as long as
|
24
|
+
a named queue was in a locked state.
|
25
|
+
|
26
|
+
* Add built-in job `Workhorse::Jobs::DetectStaleJobsJob` which you can schedule.
|
27
|
+
It picks up jobs that remained `locked` or `started` (running) for more than a
|
28
|
+
certain amount of time. If any of these jobs are found, an exception is thrown
|
29
|
+
(which may cause a notification if you configured `on_exception` accordingly).
|
30
|
+
See the job's API documentation for more information.
|
31
|
+
|
32
|
+
**If using oracle:** Make sure to grant execute permission to the package
|
33
|
+
`DBMS_LOCK` for your oracle database schema:
|
34
|
+
|
35
|
+
```GRANT execute ON DBMS_LOCK TO <schema-name>;```
|
36
|
+
|
3
37
|
## 0.6.9 - 2020-04-22
|
4
38
|
|
5
39
|
* Fix error where processes may have mistakenly been detected as running (add a
|
data/FAQ.md
CHANGED
@@ -74,4 +74,8 @@ production mode.
|
|
74
74
|
|
75
75
|
## Why does workhorse not support timeouts?
|
76
76
|
|
77
|
-
Generic timeout implementations are [a dangerous
|
77
|
+
Generic timeout implementations are [a dangerous
|
78
|
+
thing](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/)
|
79
|
+
in Ruby. This is why we decided against providing this feature in Workhorse and
|
80
|
+
recommend to implement the timeouts inside of your jobs - i.e. via network
|
81
|
+
timeouts.
|
data/README.md
CHANGED
@@ -66,6 +66,15 @@ What it does not do:
|
|
66
66
|
|
67
67
|
Please customize the initializer and worker script to your liking.
|
68
68
|
|
69
|
+
### Oracle
|
70
|
+
|
71
|
+
When using Oracle databases, make sure your schema has access to the package
|
72
|
+
`DBMS_LOCK`:
|
73
|
+
|
74
|
+
```
|
75
|
+
GRANT execute ON DBMS_LOCK TO <schema-name>;
|
76
|
+
```
|
77
|
+
|
69
78
|
## Queuing jobs
|
70
79
|
|
71
80
|
### Basic jobs
|
@@ -320,7 +329,6 @@ DbJob.started
|
|
320
329
|
DbJob.succeeded
|
321
330
|
DbJob.failed
|
322
331
|
```
|
323
|
-
|
324
332
|
### Resetting jobs
|
325
333
|
|
326
334
|
Jobs in a state other than `waiting` are either being processed or else already
|
@@ -352,7 +360,6 @@ Performing a reset will reset the job state to `waiting` and it will be
|
|
352
360
|
processed again. All meta fields will be reset as well. See inline documentation
|
353
361
|
of `Workhorse::DbJob#reset!` for more details.
|
354
362
|
|
355
|
-
|
356
363
|
## Using workhorse with Rails / ActiveJob
|
357
364
|
|
358
365
|
While workhorse can be used though its custom interface as documented above, it
|
@@ -373,6 +380,37 @@ jobs database on a regular interval. Workhorse provides the job
|
|
373
380
|
`Workhose::Jobs::CleanupSucceededJobs` for this purpose that cleans up all
|
374
381
|
succeeded jobs. You can run this using your scheduler in a specific interval.
|
375
382
|
|
383
|
+
## Caveats
|
384
|
+
|
385
|
+
### Errors during polling / crashed workers
|
386
|
+
|
387
|
+
Each worker process includes one thread that polls the database for jobs and
|
388
|
+
dispatches them to individual worker threads. In case of an error in the poller
|
389
|
+
(usually due to a database connection drop), the poller aborts and gracefully
|
390
|
+
shuts down the entire worker. Jobs still being processed by this worker are
|
391
|
+
attempted to be completed during this shutdown (which only works if the database
|
392
|
+
connection is still active).
|
393
|
+
|
394
|
+
This means that you should always have an external *watcher* (usually a
|
395
|
+
cronjob), that calls the `workhorse watch` command regularly. This would
|
396
|
+
automatically restart crashed worker processes.
|
397
|
+
|
398
|
+
### Stuck queues
|
399
|
+
|
400
|
+
Jobs in named queues (non-null queues) are always run sequentially. This means
|
401
|
+
that if a job in such a queue is stuck in states `locked` or `started` (i.e. due
|
402
|
+
to a database connection failure), no more jobs of this queue will be run as the
|
403
|
+
entire queue is considered locked to ensure that no jobs of the same queue run
|
404
|
+
in parallel.
|
405
|
+
|
406
|
+
For this purpose, Workhorse provides the built-in job
|
407
|
+
`Workhorse::Jobs::DetectStaleJobsJob` which you are advised schedule on a
|
408
|
+
regular basis. It picks up jobs that remained `locked` or `started` (running)
|
409
|
+
for more than a certain amount of time. If any of these jobs are found, an
|
410
|
+
exception is thrown (which may cause a notification if you configured
|
411
|
+
`on_exception` accordingly). See the job's API documentation for more
|
412
|
+
information.
|
413
|
+
|
376
414
|
## Frequently asked questions
|
377
415
|
|
378
416
|
Please consult the [FAQ](FAQ.md).
|
data/Rakefile
CHANGED
@@ -19,6 +19,7 @@ task :gemspec do
|
|
19
19
|
spec.add_development_dependency 'colorize'
|
20
20
|
spec.add_development_dependency 'benchmark-ips'
|
21
21
|
spec.add_development_dependency 'activejob'
|
22
|
+
spec.add_development_dependency 'pry'
|
22
23
|
spec.add_dependency 'activesupport'
|
23
24
|
spec.add_dependency 'activerecord'
|
24
25
|
spec.add_dependency 'schemacop', '~> 2.0'
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.
|
1
|
+
1.0.0.beta0
|
data/lib/workhorse.rb
CHANGED
@@ -46,6 +46,7 @@ require 'workhorse/worker'
|
|
46
46
|
require 'workhorse/jobs/run_rails_op'
|
47
47
|
require 'workhorse/jobs/run_active_job'
|
48
48
|
require 'workhorse/jobs/cleanup_succeeded_jobs'
|
49
|
+
require 'workhorse/jobs/detect_stale_jobs_job'
|
49
50
|
|
50
51
|
# Daemon functionality is not available on java platforms
|
51
52
|
if RUBY_PLATFORM != 'java'
|
data/lib/workhorse/db_job.rb
CHANGED
@@ -69,7 +69,14 @@ module Workhorse
|
|
69
69
|
fail "Dirty jobs can't be locked."
|
70
70
|
end
|
71
71
|
|
72
|
+
# TODO: Remove this debug output
|
73
|
+
# if Workhorse::DbJob.lock.find(id).locked_at
|
74
|
+
# puts "Already locked (with FOR UPDATE)"
|
75
|
+
# end
|
76
|
+
|
72
77
|
if locked_at
|
78
|
+
# TODO: Remove this debug output
|
79
|
+
# puts "Already locked. Job: #{self.id} Worker: #{worker_id}"
|
73
80
|
fail "Job #{id} is already locked by #{locked_by.inspect}."
|
74
81
|
end
|
75
82
|
|
@@ -0,0 +1,48 @@
|
|
1
|
+
module Workhorse::Jobs
|
2
|
+
class DetectStaleJobsJob
|
3
|
+
# Instantiates a new stale detection job.
|
4
|
+
#
|
5
|
+
# @param locked_to_started_threshold [Integer] The maximum number of seconds
|
6
|
+
# a job is allowed to stay 'locked' before this job throws an exception.
|
7
|
+
# Set this to 0 to skip this check.
|
8
|
+
# @param run_time_threshold [Integer] The maximum number of seconds
|
9
|
+
# a job is allowed to run before this job throws an exception. Set this to
|
10
|
+
# 0 to skip this check.
|
11
|
+
def initialize(locked_to_started_threshold: 3 * 60, run_time_threshold: 12 * 60)
|
12
|
+
@locked_to_started_threshold = locked_to_started_threshold
|
13
|
+
@run_time_threshold = run_time_threshold
|
14
|
+
end
|
15
|
+
|
16
|
+
def perform
|
17
|
+
messages = []
|
18
|
+
|
19
|
+
# Detect jobs that are locked for too long #
|
20
|
+
if @locked_to_started_threshold != 0
|
21
|
+
rel = Workhorse::DbJob.locked
|
22
|
+
rel = rel.where('locked_at < ?', @locked_to_started_threshold.seconds.ago)
|
23
|
+
ids = rel.pluck(:id)
|
24
|
+
|
25
|
+
if ids.size > 0
|
26
|
+
messages << "Detected #{ids.size} jobs that were locked more than "\
|
27
|
+
"#{@locked_to_started_threshold}s ago and might be stale: #{ids.inspect}."
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
# Detect jobs that are running for too long #
|
32
|
+
if @run_time_threshold != 0
|
33
|
+
rel = Workhorse::DbJob.started
|
34
|
+
rel = rel.where('started_at < ?', @run_time_threshold.seconds.ago)
|
35
|
+
ids = rel.pluck(:id)
|
36
|
+
|
37
|
+
if ids.size > 0
|
38
|
+
messages << "Detected #{ids.size} jobs that are running for longer than "\
|
39
|
+
"#{@run_time_threshold}s ago and might be stale: #{ids.inspect}."
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
if messages.any?
|
44
|
+
fail messages.join(' ')
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
48
|
+
end
|
data/lib/workhorse/poller.rb
CHANGED
@@ -1,5 +1,11 @@
|
|
1
1
|
module Workhorse
|
2
2
|
class Poller
|
3
|
+
MIN_LOCK_TIMEOUT = 0.1 # In seconds
|
4
|
+
MAX_LOCK_TIMEOUT = 1.0 # In seconds
|
5
|
+
|
6
|
+
ORACLE_LOCK_MODE = 6 # X_MODE (exclusive)
|
7
|
+
ORACLE_LOCK_HANDLE = 478564848 # Randomly chosen number
|
8
|
+
|
3
9
|
attr_reader :worker
|
4
10
|
attr_reader :table
|
5
11
|
|
@@ -20,15 +26,20 @@ module Workhorse
|
|
20
26
|
@running = true
|
21
27
|
|
22
28
|
@thread = Thread.new do
|
23
|
-
|
24
|
-
|
25
|
-
|
29
|
+
loop do
|
30
|
+
break unless running?
|
31
|
+
|
32
|
+
begin
|
26
33
|
poll
|
27
34
|
sleep
|
35
|
+
rescue Exception => e
|
36
|
+
worker.log %(Poll encountered exception:\n#{e.message}\n#{e.backtrace.join("\n")})
|
37
|
+
worker.log 'Worker shutting down...'
|
38
|
+
Workhorse.on_exception.call(e)
|
39
|
+
@running = false
|
40
|
+
worker.instance_variable_get(:@pool).shutdown
|
41
|
+
break
|
28
42
|
end
|
29
|
-
rescue Exception => e
|
30
|
-
worker.log %(Poller stopped with exception:\n#{e.message}\n#{e.backtrace.join("\n")})
|
31
|
-
Workhorse.on_exception.call(e)
|
32
43
|
end
|
33
44
|
end
|
34
45
|
end
|
@@ -61,24 +72,55 @@ module Workhorse
|
|
61
72
|
end
|
62
73
|
end
|
63
74
|
|
75
|
+
def with_global_lock(name: :workhorse, timeout: 2, &block)
|
76
|
+
if @is_oracle
|
77
|
+
result = Workhorse::DbJob.connection.select_all(
|
78
|
+
"SELECT DBMS_LOCK.REQUEST(#{ORACLE_LOCK_HANDLE}, #{ORACLE_LOCK_MODE}, #{timeout}) FROM DUAL"
|
79
|
+
).first.values.last
|
80
|
+
|
81
|
+
success = result == 0
|
82
|
+
else
|
83
|
+
result = Workhorse::DbJob.connection.select_all(
|
84
|
+
"SELECT GET_LOCK(CONCAT(DATABASE(), '_#{name}'), #{timeout})"
|
85
|
+
).first.values.last
|
86
|
+
success = result == 1
|
87
|
+
end
|
88
|
+
|
89
|
+
return unless success
|
90
|
+
|
91
|
+
yield
|
92
|
+
ensure
|
93
|
+
if success
|
94
|
+
if @is_oracle
|
95
|
+
Workhorse::DbJob.connection.execute("SELECT DBMS_LOCK.RELEASE(#{ORACLE_LOCK_HANDLE}) FROM DUAL")
|
96
|
+
else
|
97
|
+
Workhorse::DbJob.connection.execute("SELECT RELEASE_LOCK(CONCAT(DATABASE(), '_#{name}'))")
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
64
102
|
def poll
|
65
103
|
@instant_repoll.make_false
|
66
104
|
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
jobs
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
105
|
+
timeout = [MIN_LOCK_TIMEOUT, [MAX_LOCK_TIMEOUT, worker.polling_interval].min].max
|
106
|
+
|
107
|
+
with_global_lock timeout: timeout do
|
108
|
+
Workhorse.tx_callback.call do
|
109
|
+
# As we are the only thread posting into the worker pool, it is safe to
|
110
|
+
# get the number of idle threads without mutex synchronization. The
|
111
|
+
# actual number of idle workers at time of posting can only be larger
|
112
|
+
# than or equal to the number we get here.
|
113
|
+
idle = worker.idle
|
114
|
+
|
115
|
+
worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
|
116
|
+
|
117
|
+
unless idle.zero?
|
118
|
+
jobs = queued_db_jobs(idle)
|
119
|
+
jobs.each do |job|
|
120
|
+
worker.log "Marking job #{job.id} as locked", :debug
|
121
|
+
job.mark_locked!(worker.id)
|
122
|
+
worker.perform job
|
123
|
+
end
|
82
124
|
end
|
83
125
|
end
|
84
126
|
end
|
@@ -86,16 +128,6 @@ module Workhorse
|
|
86
128
|
|
87
129
|
# Returns an Array of #{Workhorse::DbJob}s that can be started
|
88
130
|
def queued_db_jobs(limit)
|
89
|
-
# ---------------------------------------------------------------
|
90
|
-
# Lock all queued jobs that are waiting
|
91
|
-
# ---------------------------------------------------------------
|
92
|
-
Workhorse::DbJob.connection.execute(
|
93
|
-
Workhorse::DbJob.select('null').where(
|
94
|
-
table[:queue].not_eq(nil)
|
95
|
-
.and(table[:state].eq(:waiting))
|
96
|
-
).lock.to_sql
|
97
|
-
)
|
98
|
-
|
99
131
|
# ---------------------------------------------------------------
|
100
132
|
# Select jobs to execute
|
101
133
|
# ---------------------------------------------------------------
|
@@ -147,20 +179,6 @@ module Workhorse
|
|
147
179
|
# Limit number of records
|
148
180
|
select = agnostic_limit(select, limit)
|
149
181
|
|
150
|
-
# Wrap the entire query in an other subselect to enable locking under
|
151
|
-
# Oracle SQL. As MySQL is able to lock the records without this additional
|
152
|
-
# complication, only do this when using the Oracle backend.
|
153
|
-
if @is_oracle
|
154
|
-
if AREL_GTE_7
|
155
|
-
select = Arel::SelectManager.new(Arel.sql('(' + select.to_sql + ')'))
|
156
|
-
else
|
157
|
-
select = Arel::SelectManager.new(ActiveRecord::Base, Arel.sql('(' + select.to_sql + ')'))
|
158
|
-
end
|
159
|
-
select = table.project(Arel.star).where(table[:id].in(select.project(:id)))
|
160
|
-
end
|
161
|
-
|
162
|
-
select = select.lock
|
163
|
-
|
164
182
|
return Workhorse::DbJob.find_by_sql(select.to_sql).to_a
|
165
183
|
end
|
166
184
|
|
@@ -214,7 +232,7 @@ module Workhorse
|
|
214
232
|
.where(table[:state].in(bad_states))
|
215
233
|
# .distinct is not chainable in older Arel versions
|
216
234
|
bad_queues_select.distinct
|
217
|
-
select = select.where(table[:queue].not_in(bad_queues_select))
|
235
|
+
select = select.where(table[:queue].not_in(bad_queues_select).or(table[:queue].eq(nil)))
|
218
236
|
|
219
237
|
# Restrict queues to valid ones as indicated by the options given to the
|
220
238
|
# worker
|
data/test/lib/test_helper.rb
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
require 'minitest/autorun'
|
2
2
|
require 'active_record'
|
3
3
|
require 'active_job'
|
4
|
+
require 'pry'
|
5
|
+
require 'colorize'
|
4
6
|
require 'mysql2'
|
5
7
|
require 'benchmark'
|
6
8
|
require 'jobs'
|
@@ -40,7 +42,14 @@ class WorkhorseTest < ActiveSupport::TestCase
|
|
40
42
|
end
|
41
43
|
end
|
42
44
|
|
43
|
-
ActiveRecord::Base.establish_connection
|
45
|
+
ActiveRecord::Base.establish_connection(
|
46
|
+
adapter: 'mysql2',
|
47
|
+
database: ENV['DB_NAME'] || 'workhorse',
|
48
|
+
username: ENV['DB_USERNAME'] || 'root',
|
49
|
+
password: ENV['DB_PASSWORD'] || '',
|
50
|
+
host: ENV['DB_HOST'] || '127.0.0.1',
|
51
|
+
pool: 10
|
52
|
+
)
|
44
53
|
|
45
54
|
require 'db_schema'
|
46
55
|
require 'workhorse'
|
@@ -48,6 +48,24 @@ class Workhorse::PollerTest < WorkhorseTest
|
|
48
48
|
assert_equal %w[q1 q2], w.poller.send(:valid_queues)
|
49
49
|
end
|
50
50
|
|
51
|
+
def test_valid_queues
|
52
|
+
w = Workhorse::Worker.new(polling_interval: 60)
|
53
|
+
|
54
|
+
assert_equal [], w.poller.send(:valid_queues)
|
55
|
+
|
56
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 2), queue: nil
|
57
|
+
|
58
|
+
assert_equal [nil], w.poller.send(:valid_queues)
|
59
|
+
|
60
|
+
a_job = Workhorse.enqueue BasicJob.new(sleep_time: 2), queue: :a
|
61
|
+
|
62
|
+
assert_equal [nil, 'a'], w.poller.send(:valid_queues)
|
63
|
+
|
64
|
+
a_job.update_attribute :state, :locked
|
65
|
+
|
66
|
+
assert_equal [nil], w.poller.send(:valid_queues)
|
67
|
+
end
|
68
|
+
|
51
69
|
def test_no_queues
|
52
70
|
w = Workhorse::Worker.new(polling_interval: 60)
|
53
71
|
assert_equal [], w.poller.send(:valid_queues)
|
@@ -96,6 +114,67 @@ class Workhorse::PollerTest < WorkhorseTest
|
|
96
114
|
assert_equal 1, Workhorse::DbJob.where(state: :succeeded).count
|
97
115
|
end
|
98
116
|
|
117
|
+
def test_already_locked_issue
|
118
|
+
# Create 100 jobs
|
119
|
+
100.times do |i|
|
120
|
+
Workhorse.enqueue BasicJob.new(some_param: i, sleep_time: 0)
|
121
|
+
end
|
122
|
+
|
123
|
+
# Create 25 worker processes that work for 10s each
|
124
|
+
25.times do
|
125
|
+
Process.fork do
|
126
|
+
work 10, pool_size: 1, polling_interval: 0.1
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
# Create additional 100 jobs that are scheduled while the workers are
|
131
|
+
# already polling (to make sure those are picked up as well)
|
132
|
+
100.times do
|
133
|
+
sleep 0.05
|
134
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 0)
|
135
|
+
end
|
136
|
+
|
137
|
+
# Wait for all forked processes to finish (should take ~10s)
|
138
|
+
Process.waitall
|
139
|
+
|
140
|
+
total = Workhorse::DbJob.count
|
141
|
+
succeeded = Workhorse::DbJob.succeeded.count
|
142
|
+
used_workers = Workhorse::DbJob.lock.pluck(:locked_by).uniq.size
|
143
|
+
|
144
|
+
# Make sure there are 200 jobs, all jobs have succeeded and that all of the
|
145
|
+
# workers have had their turn.
|
146
|
+
assert_equal 200, total
|
147
|
+
assert_equal 200, succeeded
|
148
|
+
assert_equal 25, used_workers
|
149
|
+
end
|
150
|
+
|
151
|
+
def test_connection_loss
|
152
|
+
$thread_conn = nil
|
153
|
+
|
154
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 3)
|
155
|
+
|
156
|
+
t = Thread.new do
|
157
|
+
w = Workhorse::Worker.new(pool_size: 5, polling_interval: 0.1)
|
158
|
+
w.start
|
159
|
+
|
160
|
+
sleep 0.5
|
161
|
+
|
162
|
+
w.poller.define_singleton_method :poll do
|
163
|
+
fail ActiveRecord::StatementInvalid, 'Mysql2::Error: Connection was killed'
|
164
|
+
end
|
165
|
+
|
166
|
+
w.wait
|
167
|
+
end
|
168
|
+
|
169
|
+
assert_nothing_raised do
|
170
|
+
Timeout.timeout(6) do
|
171
|
+
t.join
|
172
|
+
end
|
173
|
+
end
|
174
|
+
|
175
|
+
assert_equal 1, Workhorse::DbJob.succeeded.count
|
176
|
+
end
|
177
|
+
|
99
178
|
private
|
100
179
|
|
101
180
|
def setup
|
data/workhorse.gemspec
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
# -*- encoding: utf-8 -*-
|
2
|
-
# stub: workhorse 0.
|
2
|
+
# stub: workhorse 1.0.0.beta0 ruby lib
|
3
3
|
|
4
4
|
Gem::Specification.new do |s|
|
5
5
|
s.name = "workhorse".freeze
|
6
|
-
s.version = "0.
|
6
|
+
s.version = "1.0.0.beta0"
|
7
7
|
|
8
|
-
s.required_rubygems_version = Gem::Requirement.new("
|
8
|
+
s.required_rubygems_version = Gem::Requirement.new("> 1.3.1".freeze) if s.respond_to? :required_rubygems_version=
|
9
9
|
s.require_paths = ["lib".freeze]
|
10
10
|
s.authors = ["Sitrox".freeze]
|
11
|
-
s.date = "2020-
|
12
|
-
s.files = [".gitignore".freeze, ".releaser_config".freeze, ".rubocop.yml".freeze, ".travis.yml".freeze, "CHANGELOG.md".freeze, "FAQ.md".freeze, "Gemfile".freeze, "LICENSE".freeze, "README.md".freeze, "RUBY_VERSION".freeze, "Rakefile".freeze, "VERSION".freeze, "bin/rubocop".freeze, "lib/active_job/queue_adapters/workhorse_adapter.rb".freeze, "lib/generators/workhorse/install_generator.rb".freeze, "lib/generators/workhorse/templates/bin/workhorse.rb".freeze, "lib/generators/workhorse/templates/config/initializers/workhorse.rb".freeze, "lib/generators/workhorse/templates/create_table_jobs.rb".freeze, "lib/workhorse.rb".freeze, "lib/workhorse/daemon.rb".freeze, "lib/workhorse/daemon/shell_handler.rb".freeze, "lib/workhorse/db_job.rb".freeze, "lib/workhorse/enqueuer.rb".freeze, "lib/workhorse/jobs/cleanup_succeeded_jobs.rb".freeze, "lib/workhorse/jobs/run_active_job.rb".freeze, "lib/workhorse/jobs/run_rails_op.rb".freeze, "lib/workhorse/performer.rb".freeze, "lib/workhorse/poller.rb".freeze, "lib/workhorse/pool.rb".freeze, "lib/workhorse/scoped_env.rb".freeze, "lib/workhorse/worker.rb".freeze, "test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze, "workhorse.gemspec".freeze]
|
11
|
+
s.date = "2020-08-19"
|
12
|
+
s.files = [".gitignore".freeze, ".releaser_config".freeze, ".rubocop.yml".freeze, ".travis.yml".freeze, "CHANGELOG.md".freeze, "FAQ.md".freeze, "Gemfile".freeze, "LICENSE".freeze, "README.md".freeze, "RUBY_VERSION".freeze, "Rakefile".freeze, "VERSION".freeze, "bin/rubocop".freeze, "lib/active_job/queue_adapters/workhorse_adapter.rb".freeze, "lib/generators/workhorse/install_generator.rb".freeze, "lib/generators/workhorse/templates/bin/workhorse.rb".freeze, "lib/generators/workhorse/templates/config/initializers/workhorse.rb".freeze, "lib/generators/workhorse/templates/create_table_jobs.rb".freeze, "lib/workhorse.rb".freeze, "lib/workhorse/daemon.rb".freeze, "lib/workhorse/daemon/shell_handler.rb".freeze, "lib/workhorse/db_job.rb".freeze, "lib/workhorse/enqueuer.rb".freeze, "lib/workhorse/jobs/cleanup_succeeded_jobs.rb".freeze, "lib/workhorse/jobs/detect_stale_jobs_job.rb".freeze, "lib/workhorse/jobs/run_active_job.rb".freeze, "lib/workhorse/jobs/run_rails_op.rb".freeze, "lib/workhorse/performer.rb".freeze, "lib/workhorse/poller.rb".freeze, "lib/workhorse/pool.rb".freeze, "lib/workhorse/scoped_env.rb".freeze, "lib/workhorse/worker.rb".freeze, "test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze, "workhorse.gemspec".freeze]
|
13
13
|
s.rubygems_version = "3.0.3".freeze
|
14
14
|
s.summary = "Multi-threaded job backend with database queuing for ruby.".freeze
|
15
15
|
s.test_files = ["test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze]
|
@@ -26,6 +26,7 @@ Gem::Specification.new do |s|
|
|
26
26
|
s.add_development_dependency(%q<colorize>.freeze, [">= 0"])
|
27
27
|
s.add_development_dependency(%q<benchmark-ips>.freeze, [">= 0"])
|
28
28
|
s.add_development_dependency(%q<activejob>.freeze, [">= 0"])
|
29
|
+
s.add_development_dependency(%q<pry>.freeze, [">= 0"])
|
29
30
|
s.add_runtime_dependency(%q<activesupport>.freeze, [">= 0"])
|
30
31
|
s.add_runtime_dependency(%q<activerecord>.freeze, [">= 0"])
|
31
32
|
s.add_runtime_dependency(%q<schemacop>.freeze, ["~> 2.0"])
|
@@ -39,6 +40,7 @@ Gem::Specification.new do |s|
|
|
39
40
|
s.add_dependency(%q<colorize>.freeze, [">= 0"])
|
40
41
|
s.add_dependency(%q<benchmark-ips>.freeze, [">= 0"])
|
41
42
|
s.add_dependency(%q<activejob>.freeze, [">= 0"])
|
43
|
+
s.add_dependency(%q<pry>.freeze, [">= 0"])
|
42
44
|
s.add_dependency(%q<activesupport>.freeze, [">= 0"])
|
43
45
|
s.add_dependency(%q<activerecord>.freeze, [">= 0"])
|
44
46
|
s.add_dependency(%q<schemacop>.freeze, ["~> 2.0"])
|
@@ -53,6 +55,7 @@ Gem::Specification.new do |s|
|
|
53
55
|
s.add_dependency(%q<colorize>.freeze, [">= 0"])
|
54
56
|
s.add_dependency(%q<benchmark-ips>.freeze, [">= 0"])
|
55
57
|
s.add_dependency(%q<activejob>.freeze, [">= 0"])
|
58
|
+
s.add_dependency(%q<pry>.freeze, [">= 0"])
|
56
59
|
s.add_dependency(%q<activesupport>.freeze, [">= 0"])
|
57
60
|
s.add_dependency(%q<activerecord>.freeze, [">= 0"])
|
58
61
|
s.add_dependency(%q<schemacop>.freeze, ["~> 2.0"])
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: workhorse
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0.beta0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sitrox
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-08-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -122,6 +122,20 @@ dependencies:
|
|
122
122
|
- - ">="
|
123
123
|
- !ruby/object:Gem::Version
|
124
124
|
version: '0'
|
125
|
+
- !ruby/object:Gem::Dependency
|
126
|
+
name: pry
|
127
|
+
requirement: !ruby/object:Gem::Requirement
|
128
|
+
requirements:
|
129
|
+
- - ">="
|
130
|
+
- !ruby/object:Gem::Version
|
131
|
+
version: '0'
|
132
|
+
type: :development
|
133
|
+
prerelease: false
|
134
|
+
version_requirements: !ruby/object:Gem::Requirement
|
135
|
+
requirements:
|
136
|
+
- - ">="
|
137
|
+
- !ruby/object:Gem::Version
|
138
|
+
version: '0'
|
125
139
|
- !ruby/object:Gem::Dependency
|
126
140
|
name: activesupport
|
127
141
|
requirement: !ruby/object:Gem::Requirement
|
@@ -208,6 +222,7 @@ files:
|
|
208
222
|
- lib/workhorse/db_job.rb
|
209
223
|
- lib/workhorse/enqueuer.rb
|
210
224
|
- lib/workhorse/jobs/cleanup_succeeded_jobs.rb
|
225
|
+
- lib/workhorse/jobs/detect_stale_jobs_job.rb
|
211
226
|
- lib/workhorse/jobs/run_active_job.rb
|
212
227
|
- lib/workhorse/jobs/run_rails_op.rb
|
213
228
|
- lib/workhorse/performer.rb
|
@@ -240,9 +255,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
240
255
|
version: '0'
|
241
256
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
242
257
|
requirements:
|
243
|
-
- - "
|
258
|
+
- - ">"
|
244
259
|
- !ruby/object:Gem::Version
|
245
|
-
version:
|
260
|
+
version: 1.3.1
|
246
261
|
requirements: []
|
247
262
|
rubygems_version: 3.1.2
|
248
263
|
signing_key:
|