workhorse 0.0.2 → 0.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/FAQ.md +59 -0
- data/README.md +47 -48
- data/VERSION +1 -1
- data/lib/generators/workhorse/templates/config/initializers/workhorse.rb +5 -2
- data/lib/workhorse.rb +4 -1
- data/lib/workhorse/enqueuer.rb +19 -6
- data/lib/workhorse/poller.rb +15 -4
- data/lib/workhorse/pool.rb +7 -3
- data/lib/workhorse/worker.rb +11 -2
- data/test/lib/test_helper.rb +0 -1
- data/test/workhorse/enqueuer_test.rb +14 -4
- data/test/workhorse/performer_test.rb +2 -2
- data/test/workhorse/poller_test.rb +1 -1
- data/test/workhorse/pool_test.rb +6 -8
- data/test/workhorse/worker_test.rb +24 -14
- data/workhorse.gemspec +40 -41
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1a6e6801d5faf896ac2c0ac4fdee3b769239c475
|
4
|
+
data.tar.gz: 68d7918cc11af8be1961c807dd6230dd7158ae0a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1a7d958df73684464a58f66b8a6ceedfa464601ebf7ecb7de5a31ebd77c00ba9d71aa7d5b9581eca68567066e3e4f1696eadb46a2c2aec40d8cb2837b28ff88e
|
7
|
+
data.tar.gz: fbec1a172427cfccb678cf56ea5a688c68af5efc9040b0139e73e9445a7c27b6a0a04418f4dde89eb9891814c095e0f454349c16c2772cf2087bd024e15c4eb3
|
data/FAQ.md
ADDED
@@ -0,0 +1,59 @@
|
|
1
|
+
# Workhorse FAQ
|
2
|
+
|
3
|
+
## Why should I use workhorse over *X*?
|
4
|
+
|
5
|
+
There exist a variety of job backends for ruby,
|
6
|
+
[delayed_job](https://github.com/collectiveidea/delayed_job) probably being the
|
7
|
+
closest one to workhorse.
|
8
|
+
|
9
|
+
Some key advantages we feel workhorse has over other Gems:
|
10
|
+
|
11
|
+
- Workhorse is less than 500 lines of code at its core. The code is supposed to
|
12
|
+
be easily readable, understandable, and modifiable.
|
13
|
+
|
14
|
+
- Workhorse allows you to run multiple jobs simultaneously *in the same
|
15
|
+
process*.
|
16
|
+
This capability is what inspired the creation of workhorse in the first place.
|
17
|
+
|
18
|
+
We encourage you to have a look at the other projects as well and carefully
|
19
|
+
figure out which one best suits your needs.
|
20
|
+
|
21
|
+
## My code is not thread safe. How can I use workhorse safely?
|
22
|
+
|
23
|
+
Job code that is not thread safe can not be executed safely in multiple threads
|
24
|
+
of the same process. In these cases, set the `pool_size` to `1` and, if you
|
25
|
+
still want to execute multiple jobs simultaneously, the daemon `count` to a
|
26
|
+
number greater than `1`:
|
27
|
+
|
28
|
+
```ruby
|
29
|
+
Workhorse::Daemon::ShellHandler.run count: 5 do
|
30
|
+
Workhorse::Worker.start_and_wait(pool_size: 1)
|
31
|
+
end
|
32
|
+
```
|
33
|
+
|
34
|
+
## I'm using jRuby. How can I use the daemon handler?
|
35
|
+
|
36
|
+
As java processes in general can not be forked safely, the daemon handler
|
37
|
+
provided with this Gem does not support jRuby platforms.
|
38
|
+
|
39
|
+
If your jRuby application consists of one single application process, it is
|
40
|
+
recommended to just start the job backend in the same process:
|
41
|
+
|
42
|
+
```ruby
|
43
|
+
worker = Workhorse::Worker.new(pool_size: 5)
|
44
|
+
worker.start
|
45
|
+
```
|
46
|
+
|
47
|
+
This code is non-blocking, which means that it will run as long as the process
|
48
|
+
is up. Make sure to trap `INT` and `TERM` and call `worker.shutdown` when the
|
49
|
+
process stops.
|
50
|
+
|
51
|
+
If you have multiple application processes though, you may want to start the
|
52
|
+
worker in a separate process. For this purpose, adapt the startup script
|
53
|
+
`bin/workhorse.rb` so that it is blocking:
|
54
|
+
|
55
|
+
```ruby
|
56
|
+
Workhorse::Worker.start_and_wait(pool_size: 5)
|
57
|
+
```
|
58
|
+
|
59
|
+
This can then be started and "daemonized" using standard linux tools.
|
data/README.md
CHANGED
@@ -3,34 +3,38 @@
|
|
3
3
|
|
4
4
|
# Workhorse
|
5
5
|
|
6
|
-
**This Gem is still in an early stage of development. Please not use this in production yet.**
|
7
|
-
|
8
6
|
Multi-threaded job backend with database queuing for ruby.
|
9
7
|
|
10
8
|
## Introduction
|
11
9
|
|
12
|
-
|
10
|
+
How it works:
|
13
11
|
|
14
12
|
* Jobs are instances of classes that support the `perform` method.
|
15
13
|
* Jobs are persisted in the database using ActiveRecord.
|
16
14
|
* You can start one or more worker processes.
|
17
|
-
* Each worker is configurable as to which queue(s) it processes.
|
15
|
+
* Each worker is configurable as to which queue(s) it processes. Jobs in the
|
16
|
+
same queue never run simultaneously. Jobs with no queue can always run in
|
17
|
+
parallel.
|
18
18
|
* Each worker polls the database and spawns a number of threads to execute jobs
|
19
19
|
of different queues simultaneously.
|
20
20
|
|
21
|
-
What it
|
21
|
+
What it does not do:
|
22
22
|
|
23
|
-
* It
|
24
|
-
separate processes (unless you start multiple worker
|
25
|
-
|
23
|
+
* It does not spawn new processes on the fly. Jobs are run in separate threads
|
24
|
+
but not in separate processes (unless you manually start multiple worker
|
25
|
+
processes).
|
26
|
+
* It does not support retries, timeouts, and timed execution.
|
26
27
|
|
27
28
|
## Installation
|
28
29
|
|
29
30
|
### Requirements
|
30
31
|
|
31
32
|
* A database and table handler that properly supports row-level locking (such as
|
32
|
-
MySQL with InnoDB, PostgreSQL or Oracle).
|
33
|
-
*
|
33
|
+
MySQL with InnoDB, PostgreSQL, or Oracle).
|
34
|
+
* If you are planning on using the daemons handler:
|
35
|
+
* An operating system and file system that supports file locking.
|
36
|
+
* MRI ruby (aka "c ruby") as jRuby does not support `fork`. See the
|
37
|
+
[FAQ](FAQ.md##im-using-jruby-how-can-i-use-the-daemon-handler) for possible workarounds.
|
34
38
|
|
35
39
|
### Installing under Rails
|
36
40
|
|
@@ -40,6 +44,8 @@ What it isn't:
|
|
40
44
|
gem 'workhorse'
|
41
45
|
```
|
42
46
|
|
47
|
+
Install it using `bundle install` as usual.
|
48
|
+
|
43
49
|
2. Run the install generator:
|
44
50
|
|
45
51
|
```bash
|
@@ -48,18 +54,19 @@ What it isn't:
|
|
48
54
|
|
49
55
|
This generates:
|
50
56
|
|
51
|
-
* A database migration for creating
|
52
|
-
*
|
53
|
-
*
|
57
|
+
* A database migration for creating a table named `jobs`
|
58
|
+
* The initializer `config/initializers/workhorse.rb` for global configuration
|
59
|
+
* The daemon worker script `bin/workhorse.rb`
|
54
60
|
|
55
|
-
Please customize the
|
61
|
+
Please customize the initializer and worker script to your liking.
|
56
62
|
|
57
63
|
## Queuing jobs
|
58
64
|
|
59
65
|
### Basic jobs
|
60
66
|
|
61
67
|
Workhorse can handle any jobs that support the `perform` method and are
|
62
|
-
serializable. To queue a basic job, use the static method `Workhorse.enqueue
|
68
|
+
serializable. To queue a basic job, use the static method `Workhorse.enqueue`.
|
69
|
+
You can optionally pass a queue name.
|
63
70
|
|
64
71
|
```ruby
|
65
72
|
class MyJob
|
@@ -75,11 +82,6 @@ end
|
|
75
82
|
Workhorse.enqueue MyJob.new('John'), queue: :test
|
76
83
|
```
|
77
84
|
|
78
|
-
In the above example, we also specify a queue named `:test`. This means that
|
79
|
-
this job will never run simoultaneously with other jobs in the same queue. If no
|
80
|
-
queue is given, the job can always be executed simoultaneously with any other
|
81
|
-
job.
|
82
|
-
|
83
85
|
### RailsOps operations
|
84
86
|
|
85
87
|
Workhorse allows you to easily queue
|
@@ -87,16 +89,16 @@ Workhorse allows you to easily queue
|
|
87
89
|
method `Workhorse.enqueue_op`:
|
88
90
|
|
89
91
|
```ruby
|
90
|
-
Workhorse.
|
92
|
+
Workhorse.enqueue_op Operations::Jobs::CleanUpDatabase, { quiet: true }, queue: :maintenance
|
91
93
|
```
|
92
94
|
|
93
95
|
Params passed using the second argument will be used for operation instantiation
|
94
96
|
at job execution.
|
95
97
|
|
96
|
-
|
98
|
+
If you do not want to pass any params to the operation, just omit the second hash:
|
97
99
|
|
98
100
|
```ruby
|
99
|
-
Workhorse.
|
101
|
+
Workhorse.enqueue_op Operations::Jobs::CleanUpDatabase, queue: :maintenance
|
100
102
|
```
|
101
103
|
|
102
104
|
## Configuring and starting workers
|
@@ -105,12 +107,12 @@ Workers poll the database for new jobs and execute them in one or more threads.
|
|
105
107
|
Typically, one worker is started per process. While you can start workers
|
106
108
|
manually, either in your main application process(es) or in a separate one,
|
107
109
|
workhorse also provides you with a convenient way of starting one or multiple
|
108
|
-
worker processes as
|
110
|
+
worker processes as daemons.
|
109
111
|
|
110
112
|
### Start workers manually
|
111
113
|
|
112
|
-
Workers are created by
|
113
|
-
`Workhorse::Worker` instance
|
114
|
+
Workers are created by instantiating, configuring, and starting a new
|
115
|
+
`Workhorse::Worker` instance:
|
114
116
|
|
115
117
|
```ruby
|
116
118
|
Workhorse::Worker.start_and_wait(
|
@@ -121,38 +123,30 @@ Workhorse::Worker.start_and_wait(
|
|
121
123
|
)
|
122
124
|
```
|
123
125
|
|
124
|
-
See [code
|
125
|
-
|
126
|
-
|
126
|
+
See [code documentation](http://www.rubydoc.info/github/sitrox/workhorse/Workhorse%2FWorker:initialize)
|
127
|
+
for more information on the arguments. All arguments passed to `start_and_wait`
|
128
|
+
are passed to the initialize. All arguments passed to `start_and_wait` are
|
129
|
+
in turn passed to the initializer of `Workhorse::Worker`.
|
127
130
|
|
128
131
|
### Start workers using a daemon script
|
129
132
|
|
130
|
-
Using `Workhorse::Daemon
|
131
|
-
|
132
|
-
|
133
|
-
application process(es).
|
133
|
+
Using `Workhorse::Daemon::ShellHandler`, you can spawn one or multiple worker
|
134
|
+
processes automatically. This is useful for cases where you want the workers to
|
135
|
+
exist in separate processes as opposed to your main application process(es).
|
134
136
|
|
135
|
-
For this case, the workhorse install routine automatically creates
|
136
|
-
`bin/workhorse.rb
|
137
|
+
For this case, the workhorse install routine automatically creates the file
|
138
|
+
`bin/workhorse.rb`, which can be used to start one or more worker processes.
|
137
139
|
|
138
|
-
|
140
|
+
The script can be called as follows:
|
139
141
|
|
140
142
|
```bash
|
141
|
-
bin/workhorse.rb start
|
143
|
+
bin/workhorse.rb start|stop|status|watch|restart|usage
|
142
144
|
```
|
143
145
|
|
144
146
|
#### Background and customization
|
145
147
|
|
146
|
-
|
147
|
-
|
148
|
-
```ruby
|
149
|
-
Workhorse::Daemon::ShellHandler.run count: 5 do
|
150
|
-
# This runs as a daemon and will be started 5 times
|
151
|
-
end
|
152
|
-
```
|
153
|
-
|
154
|
-
Within this shell handler, you can now instantiate, configure and start a worker
|
155
|
-
as described under *Start workers manually*:
|
148
|
+
Within the shell handler, you can instantiate, configure, and start a worker as
|
149
|
+
described under [Start workers manually](#start-workers-manually):
|
156
150
|
|
157
151
|
```ruby
|
158
152
|
Workhorse::Daemon::ShellHandler.run count: 5 do
|
@@ -162,11 +156,16 @@ Workhorse::Daemon::ShellHandler.run count: 5 do
|
|
162
156
|
end
|
163
157
|
```
|
164
158
|
|
159
|
+
## Frequently asked questions
|
160
|
+
|
161
|
+
Please consult the [FAQ](FAQ.md).
|
162
|
+
|
165
163
|
## Roadmap
|
166
164
|
|
167
165
|
* [ ] ActiveJob integration for Rails
|
168
166
|
* [ ] Job timeouts
|
167
|
+
* [ ] Job priorities
|
169
168
|
|
170
169
|
## Copyright
|
171
170
|
|
172
|
-
Copyright
|
171
|
+
Copyright © 2017 Sitrox. See `LICENSE` for further details.
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
0.0.3
|
@@ -2,9 +2,12 @@ Workhorse.setup do |config|
|
|
2
2
|
# Set this to false in order to prevent jobs from being automatically
|
3
3
|
# wrapped into a transaction. The built-in workhorse logic will still run
|
4
4
|
# in transactions.
|
5
|
-
|
5
|
+
#
|
6
|
+
# config.perform_jobs_in_tx = true
|
6
7
|
|
7
|
-
# Enable this to specify an alternative callback for handling
|
8
|
+
# Enable and configure this to specify an alternative callback for handling
|
9
|
+
# transactions.
|
10
|
+
#
|
8
11
|
# config.tx_callback = proc do |&block|
|
9
12
|
# ActiveRecord::Base.transaction&(&block)
|
10
13
|
# end
|
data/lib/workhorse.rb
CHANGED
@@ -2,7 +2,11 @@ require 'socket'
|
|
2
2
|
require 'active_support/all'
|
3
3
|
require 'active_record'
|
4
4
|
|
5
|
+
require 'workhorse/enqueuer'
|
6
|
+
|
5
7
|
module Workhorse
|
8
|
+
extend Workhorse::Enqueuer
|
9
|
+
|
6
10
|
@set_up = false
|
7
11
|
|
8
12
|
# Returns the performer currently performing the active job. This can only be
|
@@ -28,7 +32,6 @@ module Workhorse
|
|
28
32
|
end
|
29
33
|
|
30
34
|
require 'workhorse/db_job'
|
31
|
-
require 'workhorse/enqueuer'
|
32
35
|
require 'workhorse/performer'
|
33
36
|
require 'workhorse/poller'
|
34
37
|
require 'workhorse/pool'
|
data/lib/workhorse/enqueuer.rb
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
module Workhorse
|
2
|
-
|
2
|
+
module Enqueuer
|
3
3
|
# Enqueue any object that is serializable and has a `perform` method
|
4
|
-
def
|
4
|
+
def enqueue(job, queue: nil)
|
5
5
|
return DbJob.create!(
|
6
6
|
queue: queue,
|
7
7
|
handler: Marshal.dump(job)
|
@@ -9,14 +9,27 @@ module Workhorse
|
|
9
9
|
end
|
10
10
|
|
11
11
|
# Enqueue an ActiveJob job
|
12
|
-
def
|
12
|
+
def enqueue_active_job(job)
|
13
13
|
enqueue job, queue: job.queue_name
|
14
14
|
end
|
15
15
|
|
16
16
|
# Enqueue the execution of an operation by its class and params
|
17
|
-
def
|
18
|
-
|
19
|
-
|
17
|
+
def enqueue_op(cls, *args)
|
18
|
+
case args.size
|
19
|
+
when 0
|
20
|
+
workhorse_args = {}
|
21
|
+
op_args = {}
|
22
|
+
when 1
|
23
|
+
workhorse_args = args.first
|
24
|
+
op_args = {}
|
25
|
+
when 2
|
26
|
+
workhorse_args, op_args = *args
|
27
|
+
else
|
28
|
+
fail ArgumentError, "wrong number of arguments (#{args.size + 1} for 2..3)"
|
29
|
+
end
|
30
|
+
|
31
|
+
job = Workhorse::Jobs::RunRailsOp.new(cls, op_args)
|
32
|
+
enqueue job, workhorse_args
|
20
33
|
end
|
21
34
|
end
|
22
35
|
end
|
data/lib/workhorse/poller.rb
CHANGED
@@ -51,7 +51,12 @@ module Workhorse
|
|
51
51
|
|
52
52
|
def poll
|
53
53
|
Workhorse.tx_callback.call do
|
54
|
+
# As we are the only thread posting into the worker pool, it is safe to
|
55
|
+
# get the number of idle threads without mutex synchronization. The
|
56
|
+
# actual number of idle workers at time of posting can only be larger
|
57
|
+
# than or equal to the number we get here.
|
54
58
|
idle = worker.idle
|
59
|
+
|
55
60
|
worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
|
56
61
|
|
57
62
|
unless idle.zero?
|
@@ -70,7 +75,7 @@ module Workhorse
|
|
70
75
|
is_oracle = ActiveRecord::Base.connection.adapter_name == 'OracleEnhanced'
|
71
76
|
|
72
77
|
# ---------------------------------------------------------------
|
73
|
-
# Lock all queued jobs that are
|
78
|
+
# Lock all queued jobs that are waiting
|
74
79
|
# ---------------------------------------------------------------
|
75
80
|
Workhorse::DbJob.connection.execute(
|
76
81
|
Workhorse::DbJob.select('null').where(
|
@@ -93,11 +98,17 @@ module Workhorse
|
|
93
98
|
select = select.where(table[:queue].not_in(bad_queries_select))
|
94
99
|
|
95
100
|
# Restrict queues to "open" ones
|
96
|
-
|
97
|
-
where = table[:queue].in(worker.queues.reject(&:nil?))
|
101
|
+
unless worker.queues.empty?
|
98
102
|
if worker.queues.include?(nil)
|
99
|
-
where =
|
103
|
+
where = table[:queue].eq(nil)
|
104
|
+
remaining_queues = worker.queues.reject(&:nil?)
|
105
|
+
unless remaining_queues.empty?
|
106
|
+
where = where.or(table[:queue].in(remaining_queues))
|
107
|
+
end
|
108
|
+
else
|
109
|
+
where = table[:queue].in(worker.queues)
|
100
110
|
end
|
111
|
+
|
101
112
|
select = select.where(where)
|
102
113
|
end
|
103
114
|
|
data/lib/workhorse/pool.rb
CHANGED
@@ -19,7 +19,7 @@ module Workhorse
|
|
19
19
|
# Posts a new work unit to the pool.
|
20
20
|
def post
|
21
21
|
mutex.synchronize do
|
22
|
-
if
|
22
|
+
if idle.zero?
|
23
23
|
fail 'All threads are busy.'
|
24
24
|
end
|
25
25
|
|
@@ -42,10 +42,14 @@ module Workhorse
|
|
42
42
|
@size - @active_threads.value
|
43
43
|
end
|
44
44
|
|
45
|
-
|
45
|
+
def wait
|
46
|
+
@executor.wait_for_termination
|
47
|
+
end
|
48
|
+
|
49
|
+
# Shuts down the pool and waits for termination.
|
46
50
|
def shutdown
|
47
51
|
@executor.shutdown
|
48
|
-
|
52
|
+
wait
|
49
53
|
end
|
50
54
|
end
|
51
55
|
end
|
data/lib/workhorse/worker.rb
CHANGED
@@ -88,9 +88,13 @@ module Workhorse
|
|
88
88
|
# properly finished before this method returns. Subsequent calls to this
|
89
89
|
# method are ignored.
|
90
90
|
def shutdown
|
91
|
+
# This is safe to be checked outside of the mutex as 'shutdown' is the
|
92
|
+
# final state this worker can be in.
|
93
|
+
return if @state == :shutdown
|
94
|
+
|
91
95
|
mutex.synchronize do
|
92
|
-
return if @state == :shutdown
|
93
96
|
assert_state! :running
|
97
|
+
|
94
98
|
log 'Shutting down'
|
95
99
|
@state = :shutdown
|
96
100
|
|
@@ -107,6 +111,7 @@ module Workhorse
|
|
107
111
|
def wait
|
108
112
|
assert_state! :running
|
109
113
|
@poller.wait
|
114
|
+
@pool.wait
|
110
115
|
end
|
111
116
|
|
112
117
|
def idle
|
@@ -133,10 +138,14 @@ module Workhorse
|
|
133
138
|
def trap_termination
|
134
139
|
SHUTDOWN_SIGNALS.each do |signal|
|
135
140
|
Signal.trap(signal) do
|
141
|
+
# Start a new thread as certain functionality (such as logging) is not
|
142
|
+
# available from within a trap context. As "shutdown" terminates
|
143
|
+
# quickly when called multiple times, this does not pose a risk of
|
144
|
+
# keeping open a big number of "shutdown threads".
|
136
145
|
Thread.new do
|
137
146
|
log "\nCaught #{signal}, shutting worker down..."
|
138
147
|
shutdown
|
139
|
-
end
|
148
|
+
end.join
|
140
149
|
end
|
141
150
|
end
|
142
151
|
end
|
data/test/lib/test_helper.rb
CHANGED
@@ -22,7 +22,6 @@ class WorkhorseTest < ActiveSupport::TestCase
|
|
22
22
|
end
|
23
23
|
end
|
24
24
|
|
25
|
-
ActiveRecord::Base.logger = Logger.new('debug.log')
|
26
25
|
ActiveRecord::Base.establish_connection adapter: 'mysql2', database: 'workhorse', username: 'travis', password: '', pool: 10, host: :localhost
|
27
26
|
|
28
27
|
require 'db_schema'
|
@@ -3,7 +3,7 @@ require 'test_helper'
|
|
3
3
|
class Workhorse::EnqueuerTest < WorkhorseTest
|
4
4
|
def test_basic
|
5
5
|
assert_equal 0, Workhorse::DbJob.all.count
|
6
|
-
Workhorse
|
6
|
+
Workhorse.enqueue BasicJob.new
|
7
7
|
assert_equal 1, Workhorse::DbJob.all.count
|
8
8
|
|
9
9
|
db_job = Workhorse::DbJob.first
|
@@ -20,7 +20,7 @@ class Workhorse::EnqueuerTest < WorkhorseTest
|
|
20
20
|
|
21
21
|
def test_with_queue
|
22
22
|
assert_equal 0, Workhorse::DbJob.all.count
|
23
|
-
Workhorse
|
23
|
+
Workhorse.enqueue BasicJob.new, queue: :q1
|
24
24
|
assert_equal 1, Workhorse::DbJob.all.count
|
25
25
|
|
26
26
|
db_job = Workhorse::DbJob.first
|
@@ -28,15 +28,25 @@ class Workhorse::EnqueuerTest < WorkhorseTest
|
|
28
28
|
end
|
29
29
|
|
30
30
|
def test_op
|
31
|
-
Workhorse
|
31
|
+
Workhorse.enqueue_op DummyRailsOpsOp, { queue: :q1 }, foo: :bar
|
32
32
|
|
33
33
|
w = Workhorse::Worker.new(queues: [:q1])
|
34
34
|
w.start
|
35
|
-
sleep 1
|
35
|
+
sleep 1
|
36
36
|
w.shutdown
|
37
37
|
|
38
38
|
assert_equal 'succeeded', Workhorse::DbJob.first.state
|
39
39
|
|
40
40
|
assert_equal [{ foo: :bar }], DummyRailsOpsOp.results
|
41
41
|
end
|
42
|
+
|
43
|
+
def test_op_without_params
|
44
|
+
Workhorse.enqueue_op DummyRailsOpsOp, queue: :q1
|
45
|
+
assert_equal 'q1', Workhorse::DbJob.first.queue
|
46
|
+
end
|
47
|
+
|
48
|
+
def test_op_without_params_and_queue
|
49
|
+
Workhorse.enqueue_op DummyRailsOpsOp
|
50
|
+
assert_nil Workhorse::DbJob.first.queue
|
51
|
+
end
|
42
52
|
end
|
@@ -6,10 +6,10 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
6
6
|
def test_db_connections
|
7
7
|
w = Workhorse::Worker.new polling_interval: 1, pool_size: 5
|
8
8
|
5.times do
|
9
|
-
Workhorse
|
9
|
+
Workhorse.enqueue DbConnectionTestJob.new
|
10
10
|
end
|
11
11
|
w.start
|
12
|
-
sleep
|
12
|
+
sleep 1
|
13
13
|
w.shutdown
|
14
14
|
|
15
15
|
assert_equal 5, DbConnectionTestJob.db_connections.count
|
data/test/workhorse/pool_test.rb
CHANGED
@@ -7,21 +7,21 @@ class Workhorse::PoolTest < WorkhorseTest
|
|
7
7
|
|
8
8
|
4.times do |_i|
|
9
9
|
p.post do
|
10
|
-
sleep
|
10
|
+
sleep 0.5
|
11
11
|
end
|
12
12
|
end
|
13
13
|
|
14
|
-
sleep 0.
|
14
|
+
sleep 0.1
|
15
15
|
assert_equal 1, p.idle
|
16
16
|
|
17
|
-
sleep
|
17
|
+
sleep 0.5
|
18
18
|
assert_equal 5, p.idle
|
19
19
|
end
|
20
20
|
end
|
21
21
|
|
22
22
|
def test_overflow
|
23
23
|
with_pool 5 do |p|
|
24
|
-
5.times { p.post { sleep
|
24
|
+
5.times { p.post { sleep 0.5 } }
|
25
25
|
|
26
26
|
exception = assert_raises do
|
27
27
|
p.post { sleep 1 }
|
@@ -37,23 +37,21 @@ class Workhorse::PoolTest < WorkhorseTest
|
|
37
37
|
|
38
38
|
5.times do
|
39
39
|
p.post do
|
40
|
-
sleep 1
|
41
40
|
counter.increment
|
42
41
|
end
|
43
42
|
end
|
44
43
|
|
45
|
-
sleep
|
44
|
+
sleep 0.01
|
46
45
|
|
47
46
|
assert_equal 5, counter.value
|
48
47
|
|
49
48
|
2.times do
|
50
49
|
p.post do
|
51
|
-
sleep 1
|
52
50
|
counter.increment
|
53
51
|
end
|
54
52
|
end
|
55
53
|
|
56
|
-
sleep
|
54
|
+
sleep 0.01
|
57
55
|
|
58
56
|
assert_equal 7, counter.value
|
59
57
|
end
|
@@ -6,12 +6,12 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
6
6
|
w.start
|
7
7
|
assert_equal 5, w.idle
|
8
8
|
|
9
|
-
Workhorse
|
9
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 0.5)
|
10
10
|
|
11
|
-
sleep
|
11
|
+
sleep 0.1
|
12
12
|
assert_equal 4, w.idle
|
13
13
|
|
14
|
-
sleep
|
14
|
+
sleep 0.5
|
15
15
|
assert_equal 5, w.idle
|
16
16
|
|
17
17
|
w.shutdown
|
@@ -29,16 +29,16 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
29
29
|
w.shutdown
|
30
30
|
w.shutdown # Should be ignored
|
31
31
|
|
32
|
-
Workhorse
|
32
|
+
Workhorse.enqueue BasicJob.new
|
33
33
|
end
|
34
34
|
|
35
35
|
def test_perform
|
36
36
|
w = Workhorse::Worker.new polling_interval: 1
|
37
|
-
Workhorse
|
37
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 0.1)
|
38
38
|
assert_equal 'waiting', Workhorse::DbJob.first.state
|
39
39
|
|
40
40
|
w.start
|
41
|
-
sleep
|
41
|
+
sleep 1
|
42
42
|
w.shutdown
|
43
43
|
|
44
44
|
assert_equal 'succeeded', Workhorse::DbJob.first.state
|
@@ -47,10 +47,10 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
47
47
|
def test_params
|
48
48
|
BasicJob.results.clear
|
49
49
|
|
50
|
-
Workhorse
|
50
|
+
Workhorse.enqueue BasicJob.new(some_param: 5, sleep_time: 0)
|
51
51
|
w = Workhorse::Worker.new polling_interval: 1
|
52
52
|
w.start
|
53
|
-
sleep
|
53
|
+
sleep 0.5
|
54
54
|
w.shutdown
|
55
55
|
|
56
56
|
assert_equal 'succeeded', Workhorse::DbJob.first.state
|
@@ -79,17 +79,27 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
79
79
|
|
80
80
|
def test_no_queues
|
81
81
|
enqueue_in_multiple_queues
|
82
|
-
work
|
82
|
+
work 1
|
83
83
|
|
84
84
|
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
85
85
|
assert_equal 'succeeded', jobs[0].state
|
86
86
|
assert_equal 'succeeded', jobs[1].state
|
87
|
+
assert_equal 'succeeded', jobs[2].state
|
88
|
+
end
|
89
|
+
|
90
|
+
def test_nil_queue
|
91
|
+
enqueue_in_multiple_queues
|
92
|
+
work 1, queues: [nil]
|
93
|
+
|
94
|
+
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
95
|
+
assert_equal 'succeeded', jobs[0].state
|
96
|
+
assert_equal 'waiting', jobs[1].state
|
87
97
|
assert_equal 'waiting', jobs[2].state
|
88
98
|
end
|
89
99
|
|
90
100
|
def test_queues_with_nil
|
91
101
|
enqueue_in_multiple_queues
|
92
|
-
work
|
102
|
+
work 1, queues: [nil, :q1]
|
93
103
|
|
94
104
|
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
95
105
|
assert_equal 'succeeded', jobs[0].state
|
@@ -99,7 +109,7 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
99
109
|
|
100
110
|
def test_queues_without_nil
|
101
111
|
enqueue_in_multiple_queues
|
102
|
-
work
|
112
|
+
work 1, queues: %i[q1 q2]
|
103
113
|
|
104
114
|
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
105
115
|
assert_equal 'waiting', jobs[0].state
|
@@ -110,8 +120,8 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
110
120
|
private
|
111
121
|
|
112
122
|
def enqueue_in_multiple_queues
|
113
|
-
Workhorse
|
114
|
-
Workhorse
|
115
|
-
Workhorse
|
123
|
+
Workhorse.enqueue BasicJob.new(some_param: nil)
|
124
|
+
Workhorse.enqueue BasicJob.new(some_param: :q1), queue: :q1
|
125
|
+
Workhorse.enqueue BasicJob.new(some_param: :q2), queue: :q2
|
116
126
|
end
|
117
127
|
end
|
data/workhorse.gemspec
CHANGED
@@ -1,55 +1,54 @@
|
|
1
1
|
# -*- encoding: utf-8 -*-
|
2
|
-
# stub: workhorse 0.0.2 ruby lib
|
3
2
|
|
4
3
|
Gem::Specification.new do |s|
|
5
|
-
s.name = "workhorse"
|
6
|
-
s.version = "0.0.
|
4
|
+
s.name = "workhorse"
|
5
|
+
s.version = "0.0.3"
|
7
6
|
|
8
|
-
s.required_rubygems_version = Gem::Requirement.new(">= 0"
|
9
|
-
s.
|
10
|
-
s.
|
11
|
-
s.
|
12
|
-
s.
|
13
|
-
s.rubygems_version = "2.
|
14
|
-
s.summary = "Multi-threaded job backend with database queuing for ruby."
|
15
|
-
s.test_files = ["test/lib/db_schema.rb"
|
7
|
+
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
8
|
+
s.authors = ["Sitrox"]
|
9
|
+
s.date = "2017-12-18"
|
10
|
+
s.files = [".gitignore", ".releaser_config", ".rubocop.yml", ".travis.yml", "FAQ.md", "Gemfile", "LICENSE", "README.md", "RUBY_VERSION", "Rakefile", "VERSION", "bin/rubocop", "lib/generators/workhorse/install_generator.rb", "lib/generators/workhorse/templates/bin/workhorse.rb", "lib/generators/workhorse/templates/config/initializers/workhorse.rb", "lib/generators/workhorse/templates/create_table_jobs.rb", "lib/workhorse.rb", "lib/workhorse/daemon.rb", "lib/workhorse/daemon/shell_handler.rb", "lib/workhorse/db_job.rb", "lib/workhorse/enqueuer.rb", "lib/workhorse/jobs/run_rails_op.rb", "lib/workhorse/performer.rb", "lib/workhorse/poller.rb", "lib/workhorse/pool.rb", "lib/workhorse/worker.rb", "test/lib/db_schema.rb", "test/lib/jobs.rb", "test/lib/test_helper.rb", "test/workhorse/enqueuer_test.rb", "test/workhorse/performer_test.rb", "test/workhorse/poller_test.rb", "test/workhorse/pool_test.rb", "test/workhorse/worker_test.rb", "workhorse.gemspec"]
|
11
|
+
s.require_paths = ["lib"]
|
12
|
+
s.rubygems_version = "2.0.14.1"
|
13
|
+
s.summary = "Multi-threaded job backend with database queuing for ruby."
|
14
|
+
s.test_files = ["test/lib/db_schema.rb", "test/lib/jobs.rb", "test/lib/test_helper.rb", "test/workhorse/enqueuer_test.rb", "test/workhorse/performer_test.rb", "test/workhorse/poller_test.rb", "test/workhorse/pool_test.rb", "test/workhorse/worker_test.rb"]
|
16
15
|
|
17
16
|
if s.respond_to? :specification_version then
|
18
17
|
s.specification_version = 4
|
19
18
|
|
20
19
|
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
21
|
-
s.add_development_dependency(%q<bundler
|
22
|
-
s.add_development_dependency(%q<rake
|
23
|
-
s.add_development_dependency(%q<rubocop
|
24
|
-
s.add_development_dependency(%q<minitest
|
25
|
-
s.add_development_dependency(%q<mysql2
|
26
|
-
s.add_development_dependency(%q<benchmark-ips
|
27
|
-
s.add_runtime_dependency(%q<activesupport
|
28
|
-
s.add_runtime_dependency(%q<activerecord
|
29
|
-
s.add_runtime_dependency(%q<schemacop
|
30
|
-
s.add_runtime_dependency(%q<concurrent-ruby
|
20
|
+
s.add_development_dependency(%q<bundler>, ["~> 1.3"])
|
21
|
+
s.add_development_dependency(%q<rake>, [">= 0"])
|
22
|
+
s.add_development_dependency(%q<rubocop>, ["= 0.51.0"])
|
23
|
+
s.add_development_dependency(%q<minitest>, [">= 0"])
|
24
|
+
s.add_development_dependency(%q<mysql2>, ["~> 0.3.13"])
|
25
|
+
s.add_development_dependency(%q<benchmark-ips>, [">= 0"])
|
26
|
+
s.add_runtime_dependency(%q<activesupport>, [">= 0"])
|
27
|
+
s.add_runtime_dependency(%q<activerecord>, [">= 0"])
|
28
|
+
s.add_runtime_dependency(%q<schemacop>, ["~> 2.0"])
|
29
|
+
s.add_runtime_dependency(%q<concurrent-ruby>, [">= 0"])
|
31
30
|
else
|
32
|
-
s.add_dependency(%q<bundler
|
33
|
-
s.add_dependency(%q<rake
|
34
|
-
s.add_dependency(%q<rubocop
|
35
|
-
s.add_dependency(%q<minitest
|
36
|
-
s.add_dependency(%q<mysql2
|
37
|
-
s.add_dependency(%q<benchmark-ips
|
38
|
-
s.add_dependency(%q<activesupport
|
39
|
-
s.add_dependency(%q<activerecord
|
40
|
-
s.add_dependency(%q<schemacop
|
41
|
-
s.add_dependency(%q<concurrent-ruby
|
31
|
+
s.add_dependency(%q<bundler>, ["~> 1.3"])
|
32
|
+
s.add_dependency(%q<rake>, [">= 0"])
|
33
|
+
s.add_dependency(%q<rubocop>, ["= 0.51.0"])
|
34
|
+
s.add_dependency(%q<minitest>, [">= 0"])
|
35
|
+
s.add_dependency(%q<mysql2>, ["~> 0.3.13"])
|
36
|
+
s.add_dependency(%q<benchmark-ips>, [">= 0"])
|
37
|
+
s.add_dependency(%q<activesupport>, [">= 0"])
|
38
|
+
s.add_dependency(%q<activerecord>, [">= 0"])
|
39
|
+
s.add_dependency(%q<schemacop>, ["~> 2.0"])
|
40
|
+
s.add_dependency(%q<concurrent-ruby>, [">= 0"])
|
42
41
|
end
|
43
42
|
else
|
44
|
-
s.add_dependency(%q<bundler
|
45
|
-
s.add_dependency(%q<rake
|
46
|
-
s.add_dependency(%q<rubocop
|
47
|
-
s.add_dependency(%q<minitest
|
48
|
-
s.add_dependency(%q<mysql2
|
49
|
-
s.add_dependency(%q<benchmark-ips
|
50
|
-
s.add_dependency(%q<activesupport
|
51
|
-
s.add_dependency(%q<activerecord
|
52
|
-
s.add_dependency(%q<schemacop
|
53
|
-
s.add_dependency(%q<concurrent-ruby
|
43
|
+
s.add_dependency(%q<bundler>, ["~> 1.3"])
|
44
|
+
s.add_dependency(%q<rake>, [">= 0"])
|
45
|
+
s.add_dependency(%q<rubocop>, ["= 0.51.0"])
|
46
|
+
s.add_dependency(%q<minitest>, [">= 0"])
|
47
|
+
s.add_dependency(%q<mysql2>, ["~> 0.3.13"])
|
48
|
+
s.add_dependency(%q<benchmark-ips>, [">= 0"])
|
49
|
+
s.add_dependency(%q<activesupport>, [">= 0"])
|
50
|
+
s.add_dependency(%q<activerecord>, [">= 0"])
|
51
|
+
s.add_dependency(%q<schemacop>, ["~> 2.0"])
|
52
|
+
s.add_dependency(%q<concurrent-ruby>, [">= 0"])
|
54
53
|
end
|
55
54
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: workhorse
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sitrox
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-12-
|
11
|
+
date: 2017-12-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -160,6 +160,7 @@ files:
|
|
160
160
|
- ".releaser_config"
|
161
161
|
- ".rubocop.yml"
|
162
162
|
- ".travis.yml"
|
163
|
+
- FAQ.md
|
163
164
|
- Gemfile
|
164
165
|
- LICENSE
|
165
166
|
- README.md
|