workhorse 0.0.2 → 0.0.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/FAQ.md +59 -0
- data/README.md +47 -48
- data/VERSION +1 -1
- data/lib/generators/workhorse/templates/config/initializers/workhorse.rb +5 -2
- data/lib/workhorse.rb +4 -1
- data/lib/workhorse/enqueuer.rb +19 -6
- data/lib/workhorse/poller.rb +15 -4
- data/lib/workhorse/pool.rb +7 -3
- data/lib/workhorse/worker.rb +11 -2
- data/test/lib/test_helper.rb +0 -1
- data/test/workhorse/enqueuer_test.rb +14 -4
- data/test/workhorse/performer_test.rb +2 -2
- data/test/workhorse/poller_test.rb +1 -1
- data/test/workhorse/pool_test.rb +6 -8
- data/test/workhorse/worker_test.rb +24 -14
- data/workhorse.gemspec +40 -41
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1a6e6801d5faf896ac2c0ac4fdee3b769239c475
|
4
|
+
data.tar.gz: 68d7918cc11af8be1961c807dd6230dd7158ae0a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1a7d958df73684464a58f66b8a6ceedfa464601ebf7ecb7de5a31ebd77c00ba9d71aa7d5b9581eca68567066e3e4f1696eadb46a2c2aec40d8cb2837b28ff88e
|
7
|
+
data.tar.gz: fbec1a172427cfccb678cf56ea5a688c68af5efc9040b0139e73e9445a7c27b6a0a04418f4dde89eb9891814c095e0f454349c16c2772cf2087bd024e15c4eb3
|
data/FAQ.md
ADDED
@@ -0,0 +1,59 @@
|
|
1
|
+
# Workhorse FAQ
|
2
|
+
|
3
|
+
## Why should I use workhorse over *X*?
|
4
|
+
|
5
|
+
There exist a variety of job backends for ruby,
|
6
|
+
[delayed_job](https://github.com/collectiveidea/delayed_job) probably being the
|
7
|
+
closest one to workhorse.
|
8
|
+
|
9
|
+
Some key advantages we feel workhorse has over other Gems:
|
10
|
+
|
11
|
+
- Workhorse is less than 500 lines of code at its core. The code is supposed to
|
12
|
+
be easily readable, understandable, and modifiable.
|
13
|
+
|
14
|
+
- Workhorse allows you to run multiple jobs simultaneously *in the same
|
15
|
+
process*.
|
16
|
+
This capability is what inspired the creation of workhorse in the first place.
|
17
|
+
|
18
|
+
We encourage you to have a look at the other projects as well and carefully
|
19
|
+
figure out which one best suits your needs.
|
20
|
+
|
21
|
+
## My code is not thread safe. How can I use workhorse safely?
|
22
|
+
|
23
|
+
Job code that is not thread safe can not be executed safely in multiple threads
|
24
|
+
of the same process. In these cases, set the `pool_size` to `1` and, if you
|
25
|
+
still want to execute multiple jobs simultaneously, the daemon `count` to a
|
26
|
+
number greater than `1`:
|
27
|
+
|
28
|
+
```ruby
|
29
|
+
Workhorse::Daemon::ShellHandler.run count: 5 do
|
30
|
+
Workhorse::Worker.start_and_wait(pool_size: 1)
|
31
|
+
end
|
32
|
+
```
|
33
|
+
|
34
|
+
## I'm using jRuby. How can I use the daemon handler?
|
35
|
+
|
36
|
+
As java processes in general can not be forked safely, the daemon handler
|
37
|
+
provided with this Gem does not support jRuby platforms.
|
38
|
+
|
39
|
+
If your jRuby application consists of one single application process, it is
|
40
|
+
recommended to just start the job backend in the same process:
|
41
|
+
|
42
|
+
```ruby
|
43
|
+
worker = Workhorse::Worker.new(pool_size: 5)
|
44
|
+
worker.start
|
45
|
+
```
|
46
|
+
|
47
|
+
This code is non-blocking, which means that it will run as long as the process
|
48
|
+
is up. Make sure to trap `INT` and `TERM` and call `worker.shutdown` when the
|
49
|
+
process stops.
|
50
|
+
|
51
|
+
If you have multiple application processes though, you may want to start the
|
52
|
+
worker in a separate process. For this purpose, adapt the startup script
|
53
|
+
`bin/workhorse.rb` so that it is blocking:
|
54
|
+
|
55
|
+
```ruby
|
56
|
+
Workhorse::Worker.start_and_wait(pool_size: 5)
|
57
|
+
```
|
58
|
+
|
59
|
+
This can then be started and "daemonized" using standard linux tools.
|
data/README.md
CHANGED
@@ -3,34 +3,38 @@
|
|
3
3
|
|
4
4
|
# Workhorse
|
5
5
|
|
6
|
-
**This Gem is still in an early stage of development. Please not use this in production yet.**
|
7
|
-
|
8
6
|
Multi-threaded job backend with database queuing for ruby.
|
9
7
|
|
10
8
|
## Introduction
|
11
9
|
|
12
|
-
|
10
|
+
How it works:
|
13
11
|
|
14
12
|
* Jobs are instances of classes that support the `perform` method.
|
15
13
|
* Jobs are persisted in the database using ActiveRecord.
|
16
14
|
* You can start one or more worker processes.
|
17
|
-
* Each worker is configurable as to which queue(s) it processes.
|
15
|
+
* Each worker is configurable as to which queue(s) it processes. Jobs in the
|
16
|
+
same queue never run simultaneously. Jobs with no queue can always run in
|
17
|
+
parallel.
|
18
18
|
* Each worker polls the database and spawns a number of threads to execute jobs
|
19
19
|
of different queues simultaneously.
|
20
20
|
|
21
|
-
What it
|
21
|
+
What it does not do:
|
22
22
|
|
23
|
-
* It
|
24
|
-
separate processes (unless you start multiple worker
|
25
|
-
|
23
|
+
* It does not spawn new processes on the fly. Jobs are run in separate threads
|
24
|
+
but not in separate processes (unless you manually start multiple worker
|
25
|
+
processes).
|
26
|
+
* It does not support retries, timeouts, and timed execution.
|
26
27
|
|
27
28
|
## Installation
|
28
29
|
|
29
30
|
### Requirements
|
30
31
|
|
31
32
|
* A database and table handler that properly supports row-level locking (such as
|
32
|
-
MySQL with InnoDB, PostgreSQL or Oracle).
|
33
|
-
*
|
33
|
+
MySQL with InnoDB, PostgreSQL, or Oracle).
|
34
|
+
* If you are planning on using the daemons handler:
|
35
|
+
* An operating system and file system that supports file locking.
|
36
|
+
* MRI ruby (aka "c ruby") as jRuby does not support `fork`. See the
|
37
|
+
[FAQ](FAQ.md##im-using-jruby-how-can-i-use-the-daemon-handler) for possible workarounds.
|
34
38
|
|
35
39
|
### Installing under Rails
|
36
40
|
|
@@ -40,6 +44,8 @@ What it isn't:
|
|
40
44
|
gem 'workhorse'
|
41
45
|
```
|
42
46
|
|
47
|
+
Install it using `bundle install` as usual.
|
48
|
+
|
43
49
|
2. Run the install generator:
|
44
50
|
|
45
51
|
```bash
|
@@ -48,18 +54,19 @@ What it isn't:
|
|
48
54
|
|
49
55
|
This generates:
|
50
56
|
|
51
|
-
* A database migration for creating
|
52
|
-
*
|
53
|
-
*
|
57
|
+
* A database migration for creating a table named `jobs`
|
58
|
+
* The initializer `config/initializers/workhorse.rb` for global configuration
|
59
|
+
* The daemon worker script `bin/workhorse.rb`
|
54
60
|
|
55
|
-
Please customize the
|
61
|
+
Please customize the initializer and worker script to your liking.
|
56
62
|
|
57
63
|
## Queuing jobs
|
58
64
|
|
59
65
|
### Basic jobs
|
60
66
|
|
61
67
|
Workhorse can handle any jobs that support the `perform` method and are
|
62
|
-
serializable. To queue a basic job, use the static method `Workhorse.enqueue
|
68
|
+
serializable. To queue a basic job, use the static method `Workhorse.enqueue`.
|
69
|
+
You can optionally pass a queue name.
|
63
70
|
|
64
71
|
```ruby
|
65
72
|
class MyJob
|
@@ -75,11 +82,6 @@ end
|
|
75
82
|
Workhorse.enqueue MyJob.new('John'), queue: :test
|
76
83
|
```
|
77
84
|
|
78
|
-
In the above example, we also specify a queue named `:test`. This means that
|
79
|
-
this job will never run simoultaneously with other jobs in the same queue. If no
|
80
|
-
queue is given, the job can always be executed simoultaneously with any other
|
81
|
-
job.
|
82
|
-
|
83
85
|
### RailsOps operations
|
84
86
|
|
85
87
|
Workhorse allows you to easily queue
|
@@ -87,16 +89,16 @@ Workhorse allows you to easily queue
|
|
87
89
|
method `Workhorse.enqueue_op`:
|
88
90
|
|
89
91
|
```ruby
|
90
|
-
Workhorse.
|
92
|
+
Workhorse.enqueue_op Operations::Jobs::CleanUpDatabase, { quiet: true }, queue: :maintenance
|
91
93
|
```
|
92
94
|
|
93
95
|
Params passed using the second argument will be used for operation instantiation
|
94
96
|
at job execution.
|
95
97
|
|
96
|
-
|
98
|
+
If you do not want to pass any params to the operation, just omit the second hash:
|
97
99
|
|
98
100
|
```ruby
|
99
|
-
Workhorse.
|
101
|
+
Workhorse.enqueue_op Operations::Jobs::CleanUpDatabase, queue: :maintenance
|
100
102
|
```
|
101
103
|
|
102
104
|
## Configuring and starting workers
|
@@ -105,12 +107,12 @@ Workers poll the database for new jobs and execute them in one or more threads.
|
|
105
107
|
Typically, one worker is started per process. While you can start workers
|
106
108
|
manually, either in your main application process(es) or in a separate one,
|
107
109
|
workhorse also provides you with a convenient way of starting one or multiple
|
108
|
-
worker processes as
|
110
|
+
worker processes as daemons.
|
109
111
|
|
110
112
|
### Start workers manually
|
111
113
|
|
112
|
-
Workers are created by
|
113
|
-
`Workhorse::Worker` instance
|
114
|
+
Workers are created by instantiating, configuring, and starting a new
|
115
|
+
`Workhorse::Worker` instance:
|
114
116
|
|
115
117
|
```ruby
|
116
118
|
Workhorse::Worker.start_and_wait(
|
@@ -121,38 +123,30 @@ Workhorse::Worker.start_and_wait(
|
|
121
123
|
)
|
122
124
|
```
|
123
125
|
|
124
|
-
See [code
|
125
|
-
|
126
|
-
|
126
|
+
See [code documentation](http://www.rubydoc.info/github/sitrox/workhorse/Workhorse%2FWorker:initialize)
|
127
|
+
for more information on the arguments. All arguments passed to `start_and_wait`
|
128
|
+
are passed to the initialize. All arguments passed to `start_and_wait` are
|
129
|
+
in turn passed to the initializer of `Workhorse::Worker`.
|
127
130
|
|
128
131
|
### Start workers using a daemon script
|
129
132
|
|
130
|
-
Using `Workhorse::Daemon
|
131
|
-
|
132
|
-
|
133
|
-
application process(es).
|
133
|
+
Using `Workhorse::Daemon::ShellHandler`, you can spawn one or multiple worker
|
134
|
+
processes automatically. This is useful for cases where you want the workers to
|
135
|
+
exist in separate processes as opposed to your main application process(es).
|
134
136
|
|
135
|
-
For this case, the workhorse install routine automatically creates
|
136
|
-
`bin/workhorse.rb
|
137
|
+
For this case, the workhorse install routine automatically creates the file
|
138
|
+
`bin/workhorse.rb`, which can be used to start one or more worker processes.
|
137
139
|
|
138
|
-
|
140
|
+
The script can be called as follows:
|
139
141
|
|
140
142
|
```bash
|
141
|
-
bin/workhorse.rb start
|
143
|
+
bin/workhorse.rb start|stop|status|watch|restart|usage
|
142
144
|
```
|
143
145
|
|
144
146
|
#### Background and customization
|
145
147
|
|
146
|
-
|
147
|
-
|
148
|
-
```ruby
|
149
|
-
Workhorse::Daemon::ShellHandler.run count: 5 do
|
150
|
-
# This runs as a daemon and will be started 5 times
|
151
|
-
end
|
152
|
-
```
|
153
|
-
|
154
|
-
Within this shell handler, you can now instantiate, configure and start a worker
|
155
|
-
as described under *Start workers manually*:
|
148
|
+
Within the shell handler, you can instantiate, configure, and start a worker as
|
149
|
+
described under [Start workers manually](#start-workers-manually):
|
156
150
|
|
157
151
|
```ruby
|
158
152
|
Workhorse::Daemon::ShellHandler.run count: 5 do
|
@@ -162,11 +156,16 @@ Workhorse::Daemon::ShellHandler.run count: 5 do
|
|
162
156
|
end
|
163
157
|
```
|
164
158
|
|
159
|
+
## Frequently asked questions
|
160
|
+
|
161
|
+
Please consult the [FAQ](FAQ.md).
|
162
|
+
|
165
163
|
## Roadmap
|
166
164
|
|
167
165
|
* [ ] ActiveJob integration for Rails
|
168
166
|
* [ ] Job timeouts
|
167
|
+
* [ ] Job priorities
|
169
168
|
|
170
169
|
## Copyright
|
171
170
|
|
172
|
-
Copyright
|
171
|
+
Copyright © 2017 Sitrox. See `LICENSE` for further details.
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
0.0.3
|
@@ -2,9 +2,12 @@ Workhorse.setup do |config|
|
|
2
2
|
# Set this to false in order to prevent jobs from being automatically
|
3
3
|
# wrapped into a transaction. The built-in workhorse logic will still run
|
4
4
|
# in transactions.
|
5
|
-
|
5
|
+
#
|
6
|
+
# config.perform_jobs_in_tx = true
|
6
7
|
|
7
|
-
# Enable this to specify an alternative callback for handling
|
8
|
+
# Enable and configure this to specify an alternative callback for handling
|
9
|
+
# transactions.
|
10
|
+
#
|
8
11
|
# config.tx_callback = proc do |&block|
|
9
12
|
# ActiveRecord::Base.transaction&(&block)
|
10
13
|
# end
|
data/lib/workhorse.rb
CHANGED
@@ -2,7 +2,11 @@ require 'socket'
|
|
2
2
|
require 'active_support/all'
|
3
3
|
require 'active_record'
|
4
4
|
|
5
|
+
require 'workhorse/enqueuer'
|
6
|
+
|
5
7
|
module Workhorse
|
8
|
+
extend Workhorse::Enqueuer
|
9
|
+
|
6
10
|
@set_up = false
|
7
11
|
|
8
12
|
# Returns the performer currently performing the active job. This can only be
|
@@ -28,7 +32,6 @@ module Workhorse
|
|
28
32
|
end
|
29
33
|
|
30
34
|
require 'workhorse/db_job'
|
31
|
-
require 'workhorse/enqueuer'
|
32
35
|
require 'workhorse/performer'
|
33
36
|
require 'workhorse/poller'
|
34
37
|
require 'workhorse/pool'
|
data/lib/workhorse/enqueuer.rb
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
module Workhorse
|
2
|
-
|
2
|
+
module Enqueuer
|
3
3
|
# Enqueue any object that is serializable and has a `perform` method
|
4
|
-
def
|
4
|
+
def enqueue(job, queue: nil)
|
5
5
|
return DbJob.create!(
|
6
6
|
queue: queue,
|
7
7
|
handler: Marshal.dump(job)
|
@@ -9,14 +9,27 @@ module Workhorse
|
|
9
9
|
end
|
10
10
|
|
11
11
|
# Enqueue an ActiveJob job
|
12
|
-
def
|
12
|
+
def enqueue_active_job(job)
|
13
13
|
enqueue job, queue: job.queue_name
|
14
14
|
end
|
15
15
|
|
16
16
|
# Enqueue the execution of an operation by its class and params
|
17
|
-
def
|
18
|
-
|
19
|
-
|
17
|
+
def enqueue_op(cls, *args)
|
18
|
+
case args.size
|
19
|
+
when 0
|
20
|
+
workhorse_args = {}
|
21
|
+
op_args = {}
|
22
|
+
when 1
|
23
|
+
workhorse_args = args.first
|
24
|
+
op_args = {}
|
25
|
+
when 2
|
26
|
+
workhorse_args, op_args = *args
|
27
|
+
else
|
28
|
+
fail ArgumentError, "wrong number of arguments (#{args.size + 1} for 2..3)"
|
29
|
+
end
|
30
|
+
|
31
|
+
job = Workhorse::Jobs::RunRailsOp.new(cls, op_args)
|
32
|
+
enqueue job, workhorse_args
|
20
33
|
end
|
21
34
|
end
|
22
35
|
end
|
data/lib/workhorse/poller.rb
CHANGED
@@ -51,7 +51,12 @@ module Workhorse
|
|
51
51
|
|
52
52
|
def poll
|
53
53
|
Workhorse.tx_callback.call do
|
54
|
+
# As we are the only thread posting into the worker pool, it is safe to
|
55
|
+
# get the number of idle threads without mutex synchronization. The
|
56
|
+
# actual number of idle workers at time of posting can only be larger
|
57
|
+
# than or equal to the number we get here.
|
54
58
|
idle = worker.idle
|
59
|
+
|
55
60
|
worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
|
56
61
|
|
57
62
|
unless idle.zero?
|
@@ -70,7 +75,7 @@ module Workhorse
|
|
70
75
|
is_oracle = ActiveRecord::Base.connection.adapter_name == 'OracleEnhanced'
|
71
76
|
|
72
77
|
# ---------------------------------------------------------------
|
73
|
-
# Lock all queued jobs that are
|
78
|
+
# Lock all queued jobs that are waiting
|
74
79
|
# ---------------------------------------------------------------
|
75
80
|
Workhorse::DbJob.connection.execute(
|
76
81
|
Workhorse::DbJob.select('null').where(
|
@@ -93,11 +98,17 @@ module Workhorse
|
|
93
98
|
select = select.where(table[:queue].not_in(bad_queries_select))
|
94
99
|
|
95
100
|
# Restrict queues to "open" ones
|
96
|
-
|
97
|
-
where = table[:queue].in(worker.queues.reject(&:nil?))
|
101
|
+
unless worker.queues.empty?
|
98
102
|
if worker.queues.include?(nil)
|
99
|
-
where =
|
103
|
+
where = table[:queue].eq(nil)
|
104
|
+
remaining_queues = worker.queues.reject(&:nil?)
|
105
|
+
unless remaining_queues.empty?
|
106
|
+
where = where.or(table[:queue].in(remaining_queues))
|
107
|
+
end
|
108
|
+
else
|
109
|
+
where = table[:queue].in(worker.queues)
|
100
110
|
end
|
111
|
+
|
101
112
|
select = select.where(where)
|
102
113
|
end
|
103
114
|
|
data/lib/workhorse/pool.rb
CHANGED
@@ -19,7 +19,7 @@ module Workhorse
|
|
19
19
|
# Posts a new work unit to the pool.
|
20
20
|
def post
|
21
21
|
mutex.synchronize do
|
22
|
-
if
|
22
|
+
if idle.zero?
|
23
23
|
fail 'All threads are busy.'
|
24
24
|
end
|
25
25
|
|
@@ -42,10 +42,14 @@ module Workhorse
|
|
42
42
|
@size - @active_threads.value
|
43
43
|
end
|
44
44
|
|
45
|
-
|
45
|
+
def wait
|
46
|
+
@executor.wait_for_termination
|
47
|
+
end
|
48
|
+
|
49
|
+
# Shuts down the pool and waits for termination.
|
46
50
|
def shutdown
|
47
51
|
@executor.shutdown
|
48
|
-
|
52
|
+
wait
|
49
53
|
end
|
50
54
|
end
|
51
55
|
end
|
data/lib/workhorse/worker.rb
CHANGED
@@ -88,9 +88,13 @@ module Workhorse
|
|
88
88
|
# properly finished before this method returns. Subsequent calls to this
|
89
89
|
# method are ignored.
|
90
90
|
def shutdown
|
91
|
+
# This is safe to be checked outside of the mutex as 'shutdown' is the
|
92
|
+
# final state this worker can be in.
|
93
|
+
return if @state == :shutdown
|
94
|
+
|
91
95
|
mutex.synchronize do
|
92
|
-
return if @state == :shutdown
|
93
96
|
assert_state! :running
|
97
|
+
|
94
98
|
log 'Shutting down'
|
95
99
|
@state = :shutdown
|
96
100
|
|
@@ -107,6 +111,7 @@ module Workhorse
|
|
107
111
|
def wait
|
108
112
|
assert_state! :running
|
109
113
|
@poller.wait
|
114
|
+
@pool.wait
|
110
115
|
end
|
111
116
|
|
112
117
|
def idle
|
@@ -133,10 +138,14 @@ module Workhorse
|
|
133
138
|
def trap_termination
|
134
139
|
SHUTDOWN_SIGNALS.each do |signal|
|
135
140
|
Signal.trap(signal) do
|
141
|
+
# Start a new thread as certain functionality (such as logging) is not
|
142
|
+
# available from within a trap context. As "shutdown" terminates
|
143
|
+
# quickly when called multiple times, this does not pose a risk of
|
144
|
+
# keeping open a big number of "shutdown threads".
|
136
145
|
Thread.new do
|
137
146
|
log "\nCaught #{signal}, shutting worker down..."
|
138
147
|
shutdown
|
139
|
-
end
|
148
|
+
end.join
|
140
149
|
end
|
141
150
|
end
|
142
151
|
end
|
data/test/lib/test_helper.rb
CHANGED
@@ -22,7 +22,6 @@ class WorkhorseTest < ActiveSupport::TestCase
|
|
22
22
|
end
|
23
23
|
end
|
24
24
|
|
25
|
-
ActiveRecord::Base.logger = Logger.new('debug.log')
|
26
25
|
ActiveRecord::Base.establish_connection adapter: 'mysql2', database: 'workhorse', username: 'travis', password: '', pool: 10, host: :localhost
|
27
26
|
|
28
27
|
require 'db_schema'
|
@@ -3,7 +3,7 @@ require 'test_helper'
|
|
3
3
|
class Workhorse::EnqueuerTest < WorkhorseTest
|
4
4
|
def test_basic
|
5
5
|
assert_equal 0, Workhorse::DbJob.all.count
|
6
|
-
Workhorse
|
6
|
+
Workhorse.enqueue BasicJob.new
|
7
7
|
assert_equal 1, Workhorse::DbJob.all.count
|
8
8
|
|
9
9
|
db_job = Workhorse::DbJob.first
|
@@ -20,7 +20,7 @@ class Workhorse::EnqueuerTest < WorkhorseTest
|
|
20
20
|
|
21
21
|
def test_with_queue
|
22
22
|
assert_equal 0, Workhorse::DbJob.all.count
|
23
|
-
Workhorse
|
23
|
+
Workhorse.enqueue BasicJob.new, queue: :q1
|
24
24
|
assert_equal 1, Workhorse::DbJob.all.count
|
25
25
|
|
26
26
|
db_job = Workhorse::DbJob.first
|
@@ -28,15 +28,25 @@ class Workhorse::EnqueuerTest < WorkhorseTest
|
|
28
28
|
end
|
29
29
|
|
30
30
|
def test_op
|
31
|
-
Workhorse
|
31
|
+
Workhorse.enqueue_op DummyRailsOpsOp, { queue: :q1 }, foo: :bar
|
32
32
|
|
33
33
|
w = Workhorse::Worker.new(queues: [:q1])
|
34
34
|
w.start
|
35
|
-
sleep 1
|
35
|
+
sleep 1
|
36
36
|
w.shutdown
|
37
37
|
|
38
38
|
assert_equal 'succeeded', Workhorse::DbJob.first.state
|
39
39
|
|
40
40
|
assert_equal [{ foo: :bar }], DummyRailsOpsOp.results
|
41
41
|
end
|
42
|
+
|
43
|
+
def test_op_without_params
|
44
|
+
Workhorse.enqueue_op DummyRailsOpsOp, queue: :q1
|
45
|
+
assert_equal 'q1', Workhorse::DbJob.first.queue
|
46
|
+
end
|
47
|
+
|
48
|
+
def test_op_without_params_and_queue
|
49
|
+
Workhorse.enqueue_op DummyRailsOpsOp
|
50
|
+
assert_nil Workhorse::DbJob.first.queue
|
51
|
+
end
|
42
52
|
end
|
@@ -6,10 +6,10 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
6
6
|
def test_db_connections
|
7
7
|
w = Workhorse::Worker.new polling_interval: 1, pool_size: 5
|
8
8
|
5.times do
|
9
|
-
Workhorse
|
9
|
+
Workhorse.enqueue DbConnectionTestJob.new
|
10
10
|
end
|
11
11
|
w.start
|
12
|
-
sleep
|
12
|
+
sleep 1
|
13
13
|
w.shutdown
|
14
14
|
|
15
15
|
assert_equal 5, DbConnectionTestJob.db_connections.count
|
data/test/workhorse/pool_test.rb
CHANGED
@@ -7,21 +7,21 @@ class Workhorse::PoolTest < WorkhorseTest
|
|
7
7
|
|
8
8
|
4.times do |_i|
|
9
9
|
p.post do
|
10
|
-
sleep
|
10
|
+
sleep 0.5
|
11
11
|
end
|
12
12
|
end
|
13
13
|
|
14
|
-
sleep 0.
|
14
|
+
sleep 0.1
|
15
15
|
assert_equal 1, p.idle
|
16
16
|
|
17
|
-
sleep
|
17
|
+
sleep 0.5
|
18
18
|
assert_equal 5, p.idle
|
19
19
|
end
|
20
20
|
end
|
21
21
|
|
22
22
|
def test_overflow
|
23
23
|
with_pool 5 do |p|
|
24
|
-
5.times { p.post { sleep
|
24
|
+
5.times { p.post { sleep 0.5 } }
|
25
25
|
|
26
26
|
exception = assert_raises do
|
27
27
|
p.post { sleep 1 }
|
@@ -37,23 +37,21 @@ class Workhorse::PoolTest < WorkhorseTest
|
|
37
37
|
|
38
38
|
5.times do
|
39
39
|
p.post do
|
40
|
-
sleep 1
|
41
40
|
counter.increment
|
42
41
|
end
|
43
42
|
end
|
44
43
|
|
45
|
-
sleep
|
44
|
+
sleep 0.01
|
46
45
|
|
47
46
|
assert_equal 5, counter.value
|
48
47
|
|
49
48
|
2.times do
|
50
49
|
p.post do
|
51
|
-
sleep 1
|
52
50
|
counter.increment
|
53
51
|
end
|
54
52
|
end
|
55
53
|
|
56
|
-
sleep
|
54
|
+
sleep 0.01
|
57
55
|
|
58
56
|
assert_equal 7, counter.value
|
59
57
|
end
|
@@ -6,12 +6,12 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
6
6
|
w.start
|
7
7
|
assert_equal 5, w.idle
|
8
8
|
|
9
|
-
Workhorse
|
9
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 0.5)
|
10
10
|
|
11
|
-
sleep
|
11
|
+
sleep 0.1
|
12
12
|
assert_equal 4, w.idle
|
13
13
|
|
14
|
-
sleep
|
14
|
+
sleep 0.5
|
15
15
|
assert_equal 5, w.idle
|
16
16
|
|
17
17
|
w.shutdown
|
@@ -29,16 +29,16 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
29
29
|
w.shutdown
|
30
30
|
w.shutdown # Should be ignored
|
31
31
|
|
32
|
-
Workhorse
|
32
|
+
Workhorse.enqueue BasicJob.new
|
33
33
|
end
|
34
34
|
|
35
35
|
def test_perform
|
36
36
|
w = Workhorse::Worker.new polling_interval: 1
|
37
|
-
Workhorse
|
37
|
+
Workhorse.enqueue BasicJob.new(sleep_time: 0.1)
|
38
38
|
assert_equal 'waiting', Workhorse::DbJob.first.state
|
39
39
|
|
40
40
|
w.start
|
41
|
-
sleep
|
41
|
+
sleep 1
|
42
42
|
w.shutdown
|
43
43
|
|
44
44
|
assert_equal 'succeeded', Workhorse::DbJob.first.state
|
@@ -47,10 +47,10 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
47
47
|
def test_params
|
48
48
|
BasicJob.results.clear
|
49
49
|
|
50
|
-
Workhorse
|
50
|
+
Workhorse.enqueue BasicJob.new(some_param: 5, sleep_time: 0)
|
51
51
|
w = Workhorse::Worker.new polling_interval: 1
|
52
52
|
w.start
|
53
|
-
sleep
|
53
|
+
sleep 0.5
|
54
54
|
w.shutdown
|
55
55
|
|
56
56
|
assert_equal 'succeeded', Workhorse::DbJob.first.state
|
@@ -79,17 +79,27 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
79
79
|
|
80
80
|
def test_no_queues
|
81
81
|
enqueue_in_multiple_queues
|
82
|
-
work
|
82
|
+
work 1
|
83
83
|
|
84
84
|
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
85
85
|
assert_equal 'succeeded', jobs[0].state
|
86
86
|
assert_equal 'succeeded', jobs[1].state
|
87
|
+
assert_equal 'succeeded', jobs[2].state
|
88
|
+
end
|
89
|
+
|
90
|
+
def test_nil_queue
|
91
|
+
enqueue_in_multiple_queues
|
92
|
+
work 1, queues: [nil]
|
93
|
+
|
94
|
+
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
95
|
+
assert_equal 'succeeded', jobs[0].state
|
96
|
+
assert_equal 'waiting', jobs[1].state
|
87
97
|
assert_equal 'waiting', jobs[2].state
|
88
98
|
end
|
89
99
|
|
90
100
|
def test_queues_with_nil
|
91
101
|
enqueue_in_multiple_queues
|
92
|
-
work
|
102
|
+
work 1, queues: [nil, :q1]
|
93
103
|
|
94
104
|
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
95
105
|
assert_equal 'succeeded', jobs[0].state
|
@@ -99,7 +109,7 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
99
109
|
|
100
110
|
def test_queues_without_nil
|
101
111
|
enqueue_in_multiple_queues
|
102
|
-
work
|
112
|
+
work 1, queues: %i[q1 q2]
|
103
113
|
|
104
114
|
jobs = Workhorse::DbJob.order(queue: :asc).to_a
|
105
115
|
assert_equal 'waiting', jobs[0].state
|
@@ -110,8 +120,8 @@ class Workhorse::WorkerTest < WorkhorseTest
|
|
110
120
|
private
|
111
121
|
|
112
122
|
def enqueue_in_multiple_queues
|
113
|
-
Workhorse
|
114
|
-
Workhorse
|
115
|
-
Workhorse
|
123
|
+
Workhorse.enqueue BasicJob.new(some_param: nil)
|
124
|
+
Workhorse.enqueue BasicJob.new(some_param: :q1), queue: :q1
|
125
|
+
Workhorse.enqueue BasicJob.new(some_param: :q2), queue: :q2
|
116
126
|
end
|
117
127
|
end
|
data/workhorse.gemspec
CHANGED
@@ -1,55 +1,54 @@
|
|
1
1
|
# -*- encoding: utf-8 -*-
|
2
|
-
# stub: workhorse 0.0.2 ruby lib
|
3
2
|
|
4
3
|
Gem::Specification.new do |s|
|
5
|
-
s.name = "workhorse"
|
6
|
-
s.version = "0.0.
|
4
|
+
s.name = "workhorse"
|
5
|
+
s.version = "0.0.3"
|
7
6
|
|
8
|
-
s.required_rubygems_version = Gem::Requirement.new(">= 0"
|
9
|
-
s.
|
10
|
-
s.
|
11
|
-
s.
|
12
|
-
s.
|
13
|
-
s.rubygems_version = "2.
|
14
|
-
s.summary = "Multi-threaded job backend with database queuing for ruby."
|
15
|
-
s.test_files = ["test/lib/db_schema.rb"
|
7
|
+
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
8
|
+
s.authors = ["Sitrox"]
|
9
|
+
s.date = "2017-12-18"
|
10
|
+
s.files = [".gitignore", ".releaser_config", ".rubocop.yml", ".travis.yml", "FAQ.md", "Gemfile", "LICENSE", "README.md", "RUBY_VERSION", "Rakefile", "VERSION", "bin/rubocop", "lib/generators/workhorse/install_generator.rb", "lib/generators/workhorse/templates/bin/workhorse.rb", "lib/generators/workhorse/templates/config/initializers/workhorse.rb", "lib/generators/workhorse/templates/create_table_jobs.rb", "lib/workhorse.rb", "lib/workhorse/daemon.rb", "lib/workhorse/daemon/shell_handler.rb", "lib/workhorse/db_job.rb", "lib/workhorse/enqueuer.rb", "lib/workhorse/jobs/run_rails_op.rb", "lib/workhorse/performer.rb", "lib/workhorse/poller.rb", "lib/workhorse/pool.rb", "lib/workhorse/worker.rb", "test/lib/db_schema.rb", "test/lib/jobs.rb", "test/lib/test_helper.rb", "test/workhorse/enqueuer_test.rb", "test/workhorse/performer_test.rb", "test/workhorse/poller_test.rb", "test/workhorse/pool_test.rb", "test/workhorse/worker_test.rb", "workhorse.gemspec"]
|
11
|
+
s.require_paths = ["lib"]
|
12
|
+
s.rubygems_version = "2.0.14.1"
|
13
|
+
s.summary = "Multi-threaded job backend with database queuing for ruby."
|
14
|
+
s.test_files = ["test/lib/db_schema.rb", "test/lib/jobs.rb", "test/lib/test_helper.rb", "test/workhorse/enqueuer_test.rb", "test/workhorse/performer_test.rb", "test/workhorse/poller_test.rb", "test/workhorse/pool_test.rb", "test/workhorse/worker_test.rb"]
|
16
15
|
|
17
16
|
if s.respond_to? :specification_version then
|
18
17
|
s.specification_version = 4
|
19
18
|
|
20
19
|
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
21
|
-
s.add_development_dependency(%q<bundler
|
22
|
-
s.add_development_dependency(%q<rake
|
23
|
-
s.add_development_dependency(%q<rubocop
|
24
|
-
s.add_development_dependency(%q<minitest
|
25
|
-
s.add_development_dependency(%q<mysql2
|
26
|
-
s.add_development_dependency(%q<benchmark-ips
|
27
|
-
s.add_runtime_dependency(%q<activesupport
|
28
|
-
s.add_runtime_dependency(%q<activerecord
|
29
|
-
s.add_runtime_dependency(%q<schemacop
|
30
|
-
s.add_runtime_dependency(%q<concurrent-ruby
|
20
|
+
s.add_development_dependency(%q<bundler>, ["~> 1.3"])
|
21
|
+
s.add_development_dependency(%q<rake>, [">= 0"])
|
22
|
+
s.add_development_dependency(%q<rubocop>, ["= 0.51.0"])
|
23
|
+
s.add_development_dependency(%q<minitest>, [">= 0"])
|
24
|
+
s.add_development_dependency(%q<mysql2>, ["~> 0.3.13"])
|
25
|
+
s.add_development_dependency(%q<benchmark-ips>, [">= 0"])
|
26
|
+
s.add_runtime_dependency(%q<activesupport>, [">= 0"])
|
27
|
+
s.add_runtime_dependency(%q<activerecord>, [">= 0"])
|
28
|
+
s.add_runtime_dependency(%q<schemacop>, ["~> 2.0"])
|
29
|
+
s.add_runtime_dependency(%q<concurrent-ruby>, [">= 0"])
|
31
30
|
else
|
32
|
-
s.add_dependency(%q<bundler
|
33
|
-
s.add_dependency(%q<rake
|
34
|
-
s.add_dependency(%q<rubocop
|
35
|
-
s.add_dependency(%q<minitest
|
36
|
-
s.add_dependency(%q<mysql2
|
37
|
-
s.add_dependency(%q<benchmark-ips
|
38
|
-
s.add_dependency(%q<activesupport
|
39
|
-
s.add_dependency(%q<activerecord
|
40
|
-
s.add_dependency(%q<schemacop
|
41
|
-
s.add_dependency(%q<concurrent-ruby
|
31
|
+
s.add_dependency(%q<bundler>, ["~> 1.3"])
|
32
|
+
s.add_dependency(%q<rake>, [">= 0"])
|
33
|
+
s.add_dependency(%q<rubocop>, ["= 0.51.0"])
|
34
|
+
s.add_dependency(%q<minitest>, [">= 0"])
|
35
|
+
s.add_dependency(%q<mysql2>, ["~> 0.3.13"])
|
36
|
+
s.add_dependency(%q<benchmark-ips>, [">= 0"])
|
37
|
+
s.add_dependency(%q<activesupport>, [">= 0"])
|
38
|
+
s.add_dependency(%q<activerecord>, [">= 0"])
|
39
|
+
s.add_dependency(%q<schemacop>, ["~> 2.0"])
|
40
|
+
s.add_dependency(%q<concurrent-ruby>, [">= 0"])
|
42
41
|
end
|
43
42
|
else
|
44
|
-
s.add_dependency(%q<bundler
|
45
|
-
s.add_dependency(%q<rake
|
46
|
-
s.add_dependency(%q<rubocop
|
47
|
-
s.add_dependency(%q<minitest
|
48
|
-
s.add_dependency(%q<mysql2
|
49
|
-
s.add_dependency(%q<benchmark-ips
|
50
|
-
s.add_dependency(%q<activesupport
|
51
|
-
s.add_dependency(%q<activerecord
|
52
|
-
s.add_dependency(%q<schemacop
|
53
|
-
s.add_dependency(%q<concurrent-ruby
|
43
|
+
s.add_dependency(%q<bundler>, ["~> 1.3"])
|
44
|
+
s.add_dependency(%q<rake>, [">= 0"])
|
45
|
+
s.add_dependency(%q<rubocop>, ["= 0.51.0"])
|
46
|
+
s.add_dependency(%q<minitest>, [">= 0"])
|
47
|
+
s.add_dependency(%q<mysql2>, ["~> 0.3.13"])
|
48
|
+
s.add_dependency(%q<benchmark-ips>, [">= 0"])
|
49
|
+
s.add_dependency(%q<activesupport>, [">= 0"])
|
50
|
+
s.add_dependency(%q<activerecord>, [">= 0"])
|
51
|
+
s.add_dependency(%q<schemacop>, ["~> 2.0"])
|
52
|
+
s.add_dependency(%q<concurrent-ruby>, [">= 0"])
|
54
53
|
end
|
55
54
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: workhorse
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sitrox
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-12-
|
11
|
+
date: 2017-12-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -160,6 +160,7 @@ files:
|
|
160
160
|
- ".releaser_config"
|
161
161
|
- ".rubocop.yml"
|
162
162
|
- ".travis.yml"
|
163
|
+
- FAQ.md
|
163
164
|
- Gemfile
|
164
165
|
- LICENSE
|
165
166
|
- README.md
|