gush 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +106 -10
- data/gush.gemspec +1 -1
- data/lib/gush/cli.rb +1 -1
- data/lib/gush/client.rb +2 -2
- data/lib/gush/job.rb +23 -23
- data/lib/gush/worker.rb +46 -25
- data/lib/gush/workflow.rb +29 -28
- data/spec/features/integration_spec.rb +72 -0
- data/spec/lib/gush/client_spec.rb +8 -15
- data/spec/lib/gush/job_spec.rb +9 -7
- data/spec/lib/gush/worker_spec.rb +6 -6
- data/spec/lib/gush/workflow_spec.rb +81 -40
- metadata +4 -4
- data/spec/features/workflows_spec.rb +0 -31
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5f2be71d1a76f021091ae90270ea5f01a2413ae0
|
4
|
+
data.tar.gz: c4160594dd6ad163239548623a7e67b29aa1727c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 271ac326d01d2dafd4c0092e3011a5d57206b988bfc7e5e62e375b913bcb2e8750440058e07a58d60663446fc7b7254f2b91a39ed6d038976995dccc3af0952b
|
7
|
+
data.tar.gz: a28e4ef3338b305523e7d7d275aa6d3e357ddfe5161aaef95e323b12268a301c8d59352f12c68c445392df613c29ae11f38079a6838006c3ca2920c32ce418ad
|
data/README.md
CHANGED
@@ -1,7 +1,12 @@
|
|
1
|
-
# Gush [](https://travis-ci.org/chaps-io/gush)
|
2
|
+
|
3
|
+
## [](https://chaps.io) proudly made by [Chaps](https://chaps.io)
|
2
4
|
|
3
5
|
Gush is a parallel workflow runner using only Redis as its message broker and Sidekiq for workers.
|
4
6
|
|
7
|
+
## Theory
|
8
|
+
|
9
|
+
Gush relies on directed acyclic graphs to store dependencies, see [Parallelizing Operations With Dependencies](https://msdn.microsoft.com/en-us/magazine/dd569760.aspx) by Stephen Toub.
|
5
10
|
## Installation
|
6
11
|
|
7
12
|
Add this line to your application's Gemfile:
|
@@ -26,9 +31,9 @@ Here is a complete example of a workflow you can create:
|
|
26
31
|
```ruby
|
27
32
|
# workflows/sample_workflow.rb
|
28
33
|
class SampleWorkflow < Gush::Workflow
|
29
|
-
def configure
|
30
|
-
run FetchJob1
|
31
|
-
run FetchJob2
|
34
|
+
def configure(url_to_fetch_from)
|
35
|
+
run FetchJob1, params: { url: url_to_fetch_from }
|
36
|
+
run FetchJob2, params: {some_flag: true, url: 'http://url.com'}
|
32
37
|
|
33
38
|
run PersistJob1, after: FetchJob1
|
34
39
|
run PersistJob2, after: FetchJob2
|
@@ -52,19 +57,62 @@ For the Workflow above, the graph will look like this:
|
|
52
57
|
|
53
58
|

|
54
59
|
|
55
|
-
|
60
|
+
|
61
|
+
#### Passing parameters to jobs
|
62
|
+
|
63
|
+
You can pass any primitive arguments into jobs while defining your workflow:
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
# workflows/sample_workflow.rb
|
67
|
+
class SampleWorkflow < Gush::Workflow
|
68
|
+
def configure
|
69
|
+
run FetchJob1, params: { url: "http://some.com/url" }
|
70
|
+
end
|
71
|
+
end
|
72
|
+
```
|
73
|
+
|
74
|
+
See below to learn how to access those params inside your job.
|
75
|
+
|
76
|
+
#### Defining jobs
|
56
77
|
|
57
78
|
Jobs are classes inheriting from `Gush::Job`:
|
58
79
|
|
59
80
|
```ruby
|
60
|
-
#workflows/sample/fetch_job1.rb
|
61
81
|
class FetchJob1 < Gush::Job
|
62
82
|
def work
|
63
83
|
# do some fetching from remote APIs
|
84
|
+
|
85
|
+
params #=> {url: "http://some.com/url"}
|
64
86
|
end
|
65
87
|
end
|
66
88
|
```
|
67
89
|
|
90
|
+
`params` method is a hash containing your (optional) parameters passed to `run` method in the workflow.
|
91
|
+
|
92
|
+
#### Passing arguments to workflows
|
93
|
+
|
94
|
+
Workflows can accept any primitive arguments in their constructor, which then will be availabe in your
|
95
|
+
`configure` method.
|
96
|
+
|
97
|
+
Here's an example of a workflow responsible for publishing a book:
|
98
|
+
|
99
|
+
```ruby
|
100
|
+
# workflows/sample_workflow.rb
|
101
|
+
class PublishBookWorkflow < Gush::Workflow
|
102
|
+
def configure(url, isbn)
|
103
|
+
run FetchBook, params: { url: url }
|
104
|
+
run PublishBook, params: { book_isbn: isbn }
|
105
|
+
end
|
106
|
+
end
|
107
|
+
```
|
108
|
+
|
109
|
+
and then create your workflow with those arguments:
|
110
|
+
|
111
|
+
```ruby
|
112
|
+
PublishBookWorkflow.new("http://url.com/book.pdf", "978-0470081204")
|
113
|
+
```
|
114
|
+
|
115
|
+
|
68
116
|
### Running workflows
|
69
117
|
|
70
118
|
Now that we have defined our workflow we can use it:
|
@@ -72,14 +120,14 @@ Now that we have defined our workflow we can use it:
|
|
72
120
|
#### 1. Initialize and save it
|
73
121
|
|
74
122
|
```ruby
|
75
|
-
flow = SampleWorkflow.new
|
123
|
+
flow = SampleWorkflow.new(optional, arguments)
|
76
124
|
flow.save # saves workflow and its jobs to Redis
|
77
125
|
```
|
78
126
|
|
79
127
|
**or:** you can also use a shortcut:
|
80
128
|
|
81
129
|
```ruby
|
82
|
-
flow = SampleWorkflow.create
|
130
|
+
flow = SampleWorkflow.create(optional, arguments)
|
83
131
|
```
|
84
132
|
|
85
133
|
#### 2. Start workflow
|
@@ -99,6 +147,50 @@ flow.start!
|
|
99
147
|
Now Gush will start processing jobs in background using Sidekiq
|
100
148
|
in the order defined in `configure` method inside Workflow.
|
101
149
|
|
150
|
+
### Pipelining
|
151
|
+
|
152
|
+
Gush offers a useful feature which lets you pass results of a job to its dependencies, so they can act accordingly.
|
153
|
+
|
154
|
+
**Example:**
|
155
|
+
|
156
|
+
Let's assume you have two jobs, `DownloadVideo`, `EncodeVideo`.
|
157
|
+
The latter needs to know where the first one downloaded the file to be able to open it.
|
158
|
+
|
159
|
+
|
160
|
+
```ruby
|
161
|
+
class DownloadVideo < Gush::Job
|
162
|
+
def work
|
163
|
+
downloader = VideoDownloader.fetch("http://youtube.com/?v=someytvideo")
|
164
|
+
|
165
|
+
output(downloader.file_path)
|
166
|
+
end
|
167
|
+
end
|
168
|
+
```
|
169
|
+
|
170
|
+
`output` method is Gush's way of saying: "I want to pass this down to my descendants".
|
171
|
+
|
172
|
+
Now, since `DownloadVideo` finished and its dependant job `EncodeVideo` started, we can access that payload down the (pipe)line:
|
173
|
+
|
174
|
+
```ruby
|
175
|
+
class EncodeVideo < Gush::Job
|
176
|
+
def work
|
177
|
+
video_path = payloads["DownloadVideo"]
|
178
|
+
end
|
179
|
+
end
|
180
|
+
```
|
181
|
+
|
182
|
+
`payloads` is a hash containing outputs from all parent jobs, where job class names are the keys.
|
183
|
+
|
184
|
+
**Note:** `payloads` will only contain outputs of the job's ancestors. So if job `A` depends on `B` and `C`,
|
185
|
+
the `paylods` hash will look like this:
|
186
|
+
|
187
|
+
```ruby
|
188
|
+
{
|
189
|
+
"B" => (...),
|
190
|
+
"C" => (...)
|
191
|
+
}
|
192
|
+
```
|
193
|
+
|
102
194
|
|
103
195
|
### Checking status:
|
104
196
|
|
@@ -126,9 +218,8 @@ flow.status
|
|
126
218
|
bundle gush list
|
127
219
|
```
|
128
220
|
|
129
|
-
### Requiring workflows inside your projects
|
130
221
|
|
131
|
-
|
222
|
+
### Requiring workflows inside your projects
|
132
223
|
|
133
224
|
When using Gush and its CLI commands you need a Gushfile.rb in root directory.
|
134
225
|
Gushfile should require all your Workflows and jobs, for example:
|
@@ -141,6 +232,11 @@ Dir[Rails.root.join("app/workflows/**/*.rb")].each do |file|
|
|
141
232
|
end
|
142
233
|
```
|
143
234
|
|
235
|
+
## Contributors
|
236
|
+
|
237
|
+
- [Mateusz Lenik](https://github.com/mlen)
|
238
|
+
- [Michał Krzyżanowski](https://github.com/krzyzak)
|
239
|
+
|
144
240
|
## Contributing
|
145
241
|
|
146
242
|
1. Fork it ( http://github.com/pokonski/gush/fork )
|
data/gush.gemspec
CHANGED
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
|
5
5
|
Gem::Specification.new do |spec|
|
6
6
|
spec.name = "gush"
|
7
|
-
spec.version = "0.1.
|
7
|
+
spec.version = "0.1.2"
|
8
8
|
spec.authors = ["Piotrek Okoński"]
|
9
9
|
spec.email = ["piotrek@okonski.org"]
|
10
10
|
spec.summary = "Fast and distributed workflow runner using only Sidekiq and Redis"
|
data/lib/gush/cli.rb
CHANGED
data/lib/gush/client.rb
CHANGED
@@ -117,7 +117,7 @@ module Gush
|
|
117
117
|
sidekiq.push(
|
118
118
|
'class' => Gush::Worker,
|
119
119
|
'queue' => configuration.namespace,
|
120
|
-
'args' => [workflow_id, job.class.to_s
|
120
|
+
'args' => [workflow_id, job.class.to_s]
|
121
121
|
)
|
122
122
|
end
|
123
123
|
|
@@ -126,7 +126,7 @@ module Gush
|
|
126
126
|
attr_reader :sidekiq, :redis
|
127
127
|
|
128
128
|
def workflow_from_hash(hash, nodes = nil)
|
129
|
-
flow = hash[:klass].constantize.new
|
129
|
+
flow = hash[:klass].constantize.new
|
130
130
|
flow.stopped = hash.fetch(:stopped, false)
|
131
131
|
flow.id = hash[:id]
|
132
132
|
|
data/lib/gush/job.rb
CHANGED
@@ -1,10 +1,8 @@
|
|
1
1
|
module Gush
|
2
2
|
class Job
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
attr_reader :name
|
3
|
+
attr_accessor :workflow_id, :incoming, :outgoing, :params,
|
4
|
+
:finished_at, :failed_at, :started_at, :enqueued_at, :payloads
|
5
|
+
attr_reader :name, :output_payload, :params
|
8
6
|
|
9
7
|
def initialize(workflow, opts = {})
|
10
8
|
@workflow = workflow
|
@@ -14,18 +12,16 @@ module Gush
|
|
14
12
|
|
15
13
|
def as_json
|
16
14
|
{
|
17
|
-
name:
|
15
|
+
name: name,
|
18
16
|
klass: self.class.to_s,
|
19
|
-
|
20
|
-
|
21
|
-
failed: failed?,
|
22
|
-
incoming: @incoming,
|
23
|
-
outgoing: @outgoing,
|
17
|
+
incoming: incoming,
|
18
|
+
outgoing: outgoing,
|
24
19
|
finished_at: finished_at,
|
25
20
|
enqueued_at: enqueued_at,
|
26
21
|
started_at: started_at,
|
27
22
|
failed_at: failed_at,
|
28
|
-
|
23
|
+
params: params,
|
24
|
+
output_payload: output_payload
|
29
25
|
}
|
30
26
|
end
|
31
27
|
|
@@ -37,6 +33,10 @@ module Gush
|
|
37
33
|
hash[:klass].constantize.new(flow, hash)
|
38
34
|
end
|
39
35
|
|
36
|
+
def output(data)
|
37
|
+
@output_payload = data
|
38
|
+
end
|
39
|
+
|
40
40
|
def work
|
41
41
|
end
|
42
42
|
|
@@ -56,8 +56,7 @@ module Gush
|
|
56
56
|
end
|
57
57
|
|
58
58
|
def fail!
|
59
|
-
@finished_at = current_timestamp
|
60
|
-
@failed_at = current_timestamp
|
59
|
+
@finished_at = @failed_at = current_timestamp
|
61
60
|
end
|
62
61
|
|
63
62
|
def enqueued?
|
@@ -89,19 +88,20 @@ module Gush
|
|
89
88
|
end
|
90
89
|
|
91
90
|
private
|
92
|
-
|
93
91
|
def current_timestamp
|
94
92
|
Time.now.to_i
|
95
93
|
end
|
96
94
|
|
97
|
-
def assign_variables(
|
98
|
-
@name
|
99
|
-
@incoming
|
100
|
-
@outgoing
|
101
|
-
@failed_at
|
102
|
-
@finished_at
|
103
|
-
@started_at
|
104
|
-
@enqueued_at
|
95
|
+
def assign_variables(opts)
|
96
|
+
@name = opts[:name]
|
97
|
+
@incoming = opts[:incoming] || []
|
98
|
+
@outgoing = opts[:outgoing] || []
|
99
|
+
@failed_at = opts[:failed_at]
|
100
|
+
@finished_at = opts[:finished_at]
|
101
|
+
@started_at = opts[:started_at]
|
102
|
+
@enqueued_at = opts[:enqueued_at]
|
103
|
+
@params = opts[:params] || {}
|
104
|
+
@output_payload = opts[:output_payload]
|
105
105
|
end
|
106
106
|
end
|
107
107
|
end
|
data/lib/gush/worker.rb
CHANGED
@@ -6,19 +6,18 @@ module Gush
|
|
6
6
|
include ::Sidekiq::Worker
|
7
7
|
sidekiq_options retry: false
|
8
8
|
|
9
|
-
def perform(workflow_id, job_id
|
10
|
-
|
9
|
+
def perform(workflow_id, job_id)
|
10
|
+
setup_job(workflow_id, job_id)
|
11
11
|
|
12
|
-
|
13
|
-
job = workflow.find_job(job_id)
|
12
|
+
job.payloads = incoming_payloads
|
14
13
|
|
15
14
|
start = Time.now
|
16
|
-
report(
|
15
|
+
report(:started, start)
|
17
16
|
|
18
17
|
failed = false
|
19
18
|
error = nil
|
20
19
|
|
21
|
-
mark_as_started
|
20
|
+
mark_as_started
|
22
21
|
begin
|
23
22
|
job.work
|
24
23
|
rescue Exception => e
|
@@ -27,46 +26,68 @@ module Gush
|
|
27
26
|
end
|
28
27
|
|
29
28
|
unless failed
|
30
|
-
report(
|
31
|
-
mark_as_finished
|
29
|
+
report(:finished, start)
|
30
|
+
mark_as_finished
|
32
31
|
|
33
|
-
enqueue_outgoing_jobs
|
32
|
+
enqueue_outgoing_jobs
|
34
33
|
else
|
35
|
-
mark_as_failed
|
36
|
-
report(
|
34
|
+
mark_as_failed
|
35
|
+
report(:failed, start, error.message)
|
37
36
|
end
|
38
37
|
end
|
39
38
|
|
40
39
|
private
|
40
|
+
attr_reader :client, :workflow, :job
|
41
41
|
|
42
|
-
|
42
|
+
def client
|
43
|
+
@client ||= Gush::Client.new(Gush.configuration)
|
44
|
+
end
|
45
|
+
|
46
|
+
def setup_job(workflow_id, job_id)
|
47
|
+
@workflow ||= client.find_workflow(workflow_id)
|
48
|
+
@job ||= workflow.find_job(job_id)
|
49
|
+
end
|
50
|
+
|
51
|
+
def incoming_payloads
|
52
|
+
payloads = {}
|
53
|
+
job.incoming.each do |job_name|
|
54
|
+
payloads[job_name] = client.load_job(workflow.id, job_name).output_payload
|
55
|
+
end
|
43
56
|
|
44
|
-
|
45
|
-
@client = Client.new(Configuration.from_json(config_json))
|
57
|
+
payloads
|
46
58
|
end
|
47
59
|
|
48
|
-
def mark_as_finished
|
60
|
+
def mark_as_finished
|
49
61
|
job.finish!
|
50
62
|
client.persist_job(workflow.id, job)
|
51
63
|
end
|
52
64
|
|
53
|
-
def mark_as_failed
|
65
|
+
def mark_as_failed
|
54
66
|
job.fail!
|
55
67
|
client.persist_job(workflow.id, job)
|
56
68
|
end
|
57
69
|
|
58
|
-
def mark_as_started
|
70
|
+
def mark_as_started
|
59
71
|
job.start!
|
60
72
|
client.persist_job(workflow.id, job)
|
61
73
|
end
|
62
74
|
|
63
|
-
def report_workflow_status
|
64
|
-
|
65
|
-
|
75
|
+
def report_workflow_status
|
76
|
+
client.workflow_report({
|
77
|
+
workflow_id: workflow.id,
|
78
|
+
status: workflow.status,
|
79
|
+
started_at: workflow.started_at,
|
80
|
+
finished_at: workflow.finished_at
|
81
|
+
})
|
66
82
|
end
|
67
83
|
|
68
|
-
def report(
|
69
|
-
message = {
|
84
|
+
def report(status, start, error = nil)
|
85
|
+
message = {
|
86
|
+
status: status,
|
87
|
+
workflow_id: workflow.id,
|
88
|
+
job: job.name,
|
89
|
+
duration: elapsed(start)
|
90
|
+
}
|
70
91
|
message[:error] = error if error
|
71
92
|
client.worker_report(message)
|
72
93
|
end
|
@@ -75,11 +96,11 @@ module Gush
|
|
75
96
|
(Time.now - start).to_f.round(3)
|
76
97
|
end
|
77
98
|
|
78
|
-
def enqueue_outgoing_jobs
|
99
|
+
def enqueue_outgoing_jobs
|
79
100
|
job.outgoing.each do |job_name|
|
80
|
-
out = client.load_job(
|
101
|
+
out = client.load_job(workflow.id, job_name)
|
81
102
|
if out.ready_to_start?
|
82
|
-
client.enqueue_job(
|
103
|
+
client.enqueue_job(workflow.id, out)
|
83
104
|
end
|
84
105
|
end
|
85
106
|
end
|
data/lib/gush/workflow.rb
CHANGED
@@ -2,19 +2,15 @@ require 'securerandom'
|
|
2
2
|
|
3
3
|
module Gush
|
4
4
|
class Workflow
|
5
|
-
attr_accessor :id, :jobs, :stopped, :persisted
|
5
|
+
attr_accessor :id, :jobs, :stopped, :persisted, :arguments
|
6
6
|
|
7
|
-
def initialize(
|
7
|
+
def initialize(*args)
|
8
8
|
@id = id
|
9
9
|
@jobs = []
|
10
10
|
@dependencies = []
|
11
11
|
@persisted = false
|
12
12
|
@stopped = false
|
13
|
-
|
14
|
-
if should_run_configure
|
15
|
-
configure
|
16
|
-
create_dependencies
|
17
|
-
end
|
13
|
+
@arguments = args
|
18
14
|
end
|
19
15
|
|
20
16
|
def self.find(id)
|
@@ -28,14 +24,12 @@ module Gush
|
|
28
24
|
end
|
29
25
|
|
30
26
|
def save
|
31
|
-
|
32
|
-
|
33
|
-
end
|
34
|
-
|
27
|
+
configure(*@arguments)
|
28
|
+
resolve_dependencies
|
35
29
|
client.persist_workflow(self)
|
36
30
|
end
|
37
31
|
|
38
|
-
def configure
|
32
|
+
def configure(*args)
|
39
33
|
end
|
40
34
|
|
41
35
|
def mark_as_stopped
|
@@ -58,7 +52,7 @@ module Gush
|
|
58
52
|
@stopped = false
|
59
53
|
end
|
60
54
|
|
61
|
-
def
|
55
|
+
def resolve_dependencies
|
62
56
|
@dependencies.each do |dependency|
|
63
57
|
from = find_job(dependency[:from])
|
64
58
|
to = find_job(dependency[:to])
|
@@ -69,7 +63,7 @@ module Gush
|
|
69
63
|
end
|
70
64
|
|
71
65
|
def find_job(name)
|
72
|
-
|
66
|
+
jobs.find { |node| node.name == name.to_s || node.class.to_s == name.to_s }
|
73
67
|
end
|
74
68
|
|
75
69
|
def finished?
|
@@ -88,23 +82,29 @@ module Gush
|
|
88
82
|
stopped
|
89
83
|
end
|
90
84
|
|
91
|
-
def run(klass,
|
92
|
-
|
93
|
-
|
85
|
+
def run(klass, opts = {})
|
86
|
+
options =
|
87
|
+
|
88
|
+
node = klass.new(self, {
|
89
|
+
name: klass.to_s,
|
90
|
+
params: opts.fetch(:params, {})
|
91
|
+
})
|
94
92
|
|
95
|
-
|
93
|
+
jobs << node
|
94
|
+
|
95
|
+
deps_after = [*opts[:after]]
|
96
96
|
deps_after.each do |dep|
|
97
97
|
@dependencies << {from: dep.to_s, to: klass.to_s }
|
98
98
|
end
|
99
99
|
|
100
|
-
deps_before = [*
|
100
|
+
deps_before = [*opts[:before]]
|
101
101
|
deps_before.each do |dep|
|
102
102
|
@dependencies << {from: klass.to_s, to: dep.to_s }
|
103
103
|
end
|
104
104
|
end
|
105
105
|
|
106
106
|
def reload
|
107
|
-
self.class.find(
|
107
|
+
self.class.find(id)
|
108
108
|
end
|
109
109
|
|
110
110
|
def initial_jobs
|
@@ -138,11 +138,12 @@ module Gush
|
|
138
138
|
name = self.class.to_s
|
139
139
|
{
|
140
140
|
name: name,
|
141
|
-
id:
|
142
|
-
|
143
|
-
|
141
|
+
id: id,
|
142
|
+
arguments: @arguments,
|
143
|
+
total: jobs.count,
|
144
|
+
finished: jobs.count(&:finished?),
|
144
145
|
klass: name,
|
145
|
-
jobs:
|
146
|
+
jobs: jobs.map(&:as_json),
|
146
147
|
status: status,
|
147
148
|
stopped: stopped,
|
148
149
|
started_at: started_at,
|
@@ -158,12 +159,12 @@ module Gush
|
|
158
159
|
ObjectSpace.each_object(Class).select { |klass| klass < self }
|
159
160
|
end
|
160
161
|
|
161
|
-
|
162
|
-
|
163
|
-
def assign_id
|
164
|
-
@id = client.next_free_id
|
162
|
+
def id
|
163
|
+
@id ||= client.next_free_id
|
165
164
|
end
|
166
165
|
|
166
|
+
private
|
167
|
+
|
167
168
|
def client
|
168
169
|
@client ||= Client.new
|
169
170
|
end
|
@@ -0,0 +1,72 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
|
4
|
+
describe "Workflows" do
|
5
|
+
it "runs the whole workflow in proper order" do
|
6
|
+
flow = TestWorkflow.create
|
7
|
+
flow.start!
|
8
|
+
|
9
|
+
expect(Gush::Worker).to have_jobs(flow.id, ["Prepare"])
|
10
|
+
|
11
|
+
Gush::Worker.perform_one
|
12
|
+
expect(Gush::Worker).to have_jobs(flow.id, ["FetchFirstJob", "FetchSecondJob"])
|
13
|
+
|
14
|
+
Gush::Worker.perform_one
|
15
|
+
expect(Gush::Worker).to have_jobs(flow.id, ["FetchSecondJob", "PersistFirstJob"])
|
16
|
+
|
17
|
+
Gush::Worker.perform_one
|
18
|
+
expect(Gush::Worker).to have_jobs(flow.id, ["PersistFirstJob", "NormalizeJob"])
|
19
|
+
|
20
|
+
Gush::Worker.perform_one
|
21
|
+
expect(Gush::Worker).to have_jobs(flow.id, ["NormalizeJob"])
|
22
|
+
|
23
|
+
Gush::Worker.perform_one
|
24
|
+
|
25
|
+
expect(Gush::Worker.jobs).to be_empty
|
26
|
+
|
27
|
+
flow = flow.reload
|
28
|
+
expect(flow).to be_finished
|
29
|
+
expect(flow).to_not be_failed
|
30
|
+
end
|
31
|
+
|
32
|
+
it "passes payloads down the workflow" do
|
33
|
+
class UpcaseJob < Gush::Job
|
34
|
+
def work
|
35
|
+
output params[:input].upcase
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
39
|
+
class PrefixJob < Gush::Job
|
40
|
+
def work
|
41
|
+
output params[:prefix].capitalize
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
class PrependJob < Gush::Job
|
46
|
+
def work
|
47
|
+
string = "#{payloads["PrefixJob"]}: #{payloads["UpcaseJob"]}"
|
48
|
+
output string
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
class PayloadWorkflow < Gush::Workflow
|
53
|
+
def configure
|
54
|
+
run UpcaseJob, params: {input: "some text"}
|
55
|
+
run PrefixJob, params: {prefix: "a prefix"}
|
56
|
+
run PrependJob, after: [UpcaseJob, PrefixJob]
|
57
|
+
end
|
58
|
+
end
|
59
|
+
|
60
|
+
flow = PayloadWorkflow.create
|
61
|
+
flow.start!
|
62
|
+
|
63
|
+
Gush::Worker.perform_one
|
64
|
+
expect(flow.reload.find_job("UpcaseJob").output_payload).to eq("SOME TEXT")
|
65
|
+
|
66
|
+
Gush::Worker.perform_one
|
67
|
+
expect(flow.reload.find_job("PrefixJob").output_payload).to eq("A prefix")
|
68
|
+
|
69
|
+
Gush::Worker.perform_one
|
70
|
+
expect(flow.reload.find_job("PrependJob").output_payload).to eq("A prefix: SOME TEXT")
|
71
|
+
end
|
72
|
+
end
|
@@ -16,8 +16,7 @@ describe Gush::Client do
|
|
16
16
|
|
17
17
|
context "when given workflow exists" do
|
18
18
|
it "returns Workflow object" do
|
19
|
-
expected_workflow = TestWorkflow.
|
20
|
-
client.persist_workflow(expected_workflow)
|
19
|
+
expected_workflow = TestWorkflow.create
|
21
20
|
workflow = client.find_workflow(expected_workflow.id)
|
22
21
|
|
23
22
|
expect(workflow.id).to eq(expected_workflow.id)
|
@@ -28,8 +27,7 @@ describe Gush::Client do
|
|
28
27
|
|
29
28
|
describe "#start_workflow" do
|
30
29
|
it "enqueues next jobs from the workflow" do
|
31
|
-
workflow = TestWorkflow.
|
32
|
-
client.persist_workflow(workflow)
|
30
|
+
workflow = TestWorkflow.create
|
33
31
|
expect {
|
34
32
|
client.start_workflow(workflow)
|
35
33
|
}.to change{Gush::Worker.jobs.count}.from(0).to(1)
|
@@ -38,15 +36,14 @@ describe Gush::Client do
|
|
38
36
|
it "removes stopped flag when the workflow is started" do
|
39
37
|
workflow = TestWorkflow.new
|
40
38
|
workflow.mark_as_stopped
|
41
|
-
|
39
|
+
workflow.save
|
42
40
|
expect {
|
43
41
|
client.start_workflow(workflow)
|
44
42
|
}.to change{client.find_workflow(workflow.id).stopped?}.from(true).to(false)
|
45
43
|
end
|
46
44
|
|
47
45
|
it "marks the enqueued jobs as enqueued" do
|
48
|
-
workflow = TestWorkflow.
|
49
|
-
client.persist_workflow(workflow)
|
46
|
+
workflow = TestWorkflow.create
|
50
47
|
client.start_workflow(workflow)
|
51
48
|
job = workflow.reload.find_job("Prepare")
|
52
49
|
expect(job.enqueued?).to eq(true)
|
@@ -55,8 +52,7 @@ describe Gush::Client do
|
|
55
52
|
|
56
53
|
describe "#stop_workflow" do
|
57
54
|
it "marks the workflow as stopped" do
|
58
|
-
workflow = TestWorkflow.
|
59
|
-
client.persist_workflow(workflow)
|
55
|
+
workflow = TestWorkflow.create
|
60
56
|
expect {
|
61
57
|
client.stop_workflow(workflow.id)
|
62
58
|
}.to change{client.find_workflow(workflow.id).stopped?}.from(false).to(true)
|
@@ -76,8 +72,7 @@ describe Gush::Client do
|
|
76
72
|
|
77
73
|
describe "#destroy_workflow" do
|
78
74
|
it "removes all Redis keys related to the workflow" do
|
79
|
-
workflow = TestWorkflow.
|
80
|
-
client.persist_workflow(workflow)
|
75
|
+
workflow = TestWorkflow.create
|
81
76
|
expect(redis.keys("gush.workflows.#{workflow.id}").length).to eq(1)
|
82
77
|
expect(redis.keys("gush.jobs.#{workflow.id}.*").length).to eq(5)
|
83
78
|
|
@@ -98,16 +93,14 @@ describe Gush::Client do
|
|
98
93
|
|
99
94
|
describe "#all_workflows" do
|
100
95
|
it "returns all registered workflows" do
|
101
|
-
workflow = TestWorkflow.
|
102
|
-
client.persist_workflow(workflow)
|
96
|
+
workflow = TestWorkflow.create
|
103
97
|
workflows = client.all_workflows
|
104
98
|
expect(workflows.map(&:id)).to eq([workflow.id])
|
105
99
|
end
|
106
100
|
end
|
107
101
|
|
108
102
|
it "should be able to handle outdated data format" do
|
109
|
-
workflow = TestWorkflow.
|
110
|
-
client.persist_workflow(workflow)
|
103
|
+
workflow = TestWorkflow.create
|
111
104
|
|
112
105
|
# malform the data
|
113
106
|
hash = Gush::JSON.decode(redis.get("gush.workflows.#{workflow.id}"), symbolize_keys: true)
|
data/spec/lib/gush/job_spec.rb
CHANGED
@@ -2,6 +2,13 @@ require 'spec_helper'
|
|
2
2
|
|
3
3
|
describe Gush::Job do
|
4
4
|
|
5
|
+
describe "#output" do
|
6
|
+
it "saves output to output_payload" do
|
7
|
+
job = described_class.new(name: "a-job")
|
8
|
+
job.output "something"
|
9
|
+
expect(job.output_payload).to eq("something")
|
10
|
+
end
|
11
|
+
end
|
5
12
|
describe "#fail!" do
|
6
13
|
it "sets finished and failed to true and records time" do
|
7
14
|
job = described_class.new(name: "a-job")
|
@@ -59,16 +66,14 @@ describe Gush::Job do
|
|
59
66
|
expected = {
|
60
67
|
name: "a-job",
|
61
68
|
klass: "Gush::Job",
|
62
|
-
finished: true,
|
63
|
-
enqueued: true,
|
64
|
-
failed: false,
|
65
69
|
incoming: [],
|
66
70
|
outgoing: [],
|
67
71
|
failed_at: nil,
|
68
72
|
started_at: nil,
|
69
73
|
finished_at: 123,
|
70
74
|
enqueued_at: 120,
|
71
|
-
|
75
|
+
params: {},
|
76
|
+
output_payload: nil
|
72
77
|
}
|
73
78
|
expect(job.as_json).to eq(expected)
|
74
79
|
end
|
@@ -82,9 +87,6 @@ describe Gush::Job do
|
|
82
87
|
{
|
83
88
|
klass: 'Gush::Job',
|
84
89
|
name: 'gob',
|
85
|
-
finished: true,
|
86
|
-
failed: true,
|
87
|
-
enqueued: true,
|
88
90
|
incoming: ['a', 'b'],
|
89
91
|
outgoing: ['c'],
|
90
92
|
failed_at: 123,
|
@@ -23,7 +23,7 @@ describe Gush::Worker do
|
|
23
23
|
allow(job).to receive(:work).and_raise(StandardError)
|
24
24
|
expect(client).to receive(:worker_report).with(hash_including(status: :failed)).ordered
|
25
25
|
|
26
|
-
subject.perform(workflow.id, "Prepare"
|
26
|
+
subject.perform(workflow.id, "Prepare")
|
27
27
|
expect(workflow.find_job("Prepare")).to be_failed
|
28
28
|
end
|
29
29
|
|
@@ -31,7 +31,7 @@ describe Gush::Worker do
|
|
31
31
|
allow(job).to receive(:work).and_raise(StandardError)
|
32
32
|
expect(client).to receive(:worker_report).with(hash_including(status: :failed)).ordered
|
33
33
|
|
34
|
-
subject.perform(workflow.id, "Prepare"
|
34
|
+
subject.perform(workflow.id, "Prepare")
|
35
35
|
end
|
36
36
|
end
|
37
37
|
|
@@ -40,13 +40,13 @@ describe Gush::Worker do
|
|
40
40
|
expect(subject).to receive(:mark_as_finished)
|
41
41
|
expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
|
42
42
|
|
43
|
-
subject.perform(workflow.id, "Prepare"
|
43
|
+
subject.perform(workflow.id, "Prepare")
|
44
44
|
end
|
45
45
|
|
46
46
|
it "reports that job succedeed" do
|
47
47
|
expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
|
48
48
|
|
49
|
-
subject.perform(workflow.id, "Prepare"
|
49
|
+
subject.perform(workflow.id, "Prepare")
|
50
50
|
end
|
51
51
|
end
|
52
52
|
|
@@ -54,14 +54,14 @@ describe Gush::Worker do
|
|
54
54
|
expect(job).to receive(:work)
|
55
55
|
expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
|
56
56
|
|
57
|
-
subject.perform(workflow.id, "Prepare"
|
57
|
+
subject.perform(workflow.id, "Prepare")
|
58
58
|
end
|
59
59
|
|
60
60
|
it "reports when the job is started" do
|
61
61
|
allow(client).to receive(:worker_report)
|
62
62
|
expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
|
63
63
|
|
64
|
-
subject.perform(workflow.id, "Prepare"
|
64
|
+
subject.perform(workflow.id, "Prepare")
|
65
65
|
end
|
66
66
|
end
|
67
67
|
end
|
@@ -1,25 +1,26 @@
|
|
1
1
|
require 'spec_helper'
|
2
2
|
|
3
3
|
describe Gush::Workflow do
|
4
|
-
subject { TestWorkflow.
|
4
|
+
subject { TestWorkflow.create }
|
5
5
|
|
6
6
|
describe "#initialize" do
|
7
|
-
|
8
|
-
it "runs #configure method " do
|
9
|
-
expect_any_instance_of(TestWorkflow).to receive(:configure)
|
10
|
-
TestWorkflow.new(true)
|
11
|
-
end
|
12
|
-
end
|
7
|
+
end
|
13
8
|
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
9
|
+
describe "#save" do
|
10
|
+
it "passes constructor arguments to the method" do
|
11
|
+
klass = Class.new(Gush::Workflow) do
|
12
|
+
def configure(*args)
|
13
|
+
run FetchFirstJob
|
14
|
+
run PersistFirstJob, after: FetchFirstJob
|
15
|
+
end
|
18
16
|
end
|
17
|
+
|
18
|
+
flow = klass.new("arg1", "arg2")
|
19
|
+
|
20
|
+
expect(flow).to receive(:configure).with("arg1", "arg2")
|
21
|
+
flow.save
|
19
22
|
end
|
20
|
-
end
|
21
23
|
|
22
|
-
describe "#save" do
|
23
24
|
context "workflow not persisted" do
|
24
25
|
it "sets persisted to true" do
|
25
26
|
flow = TestWorkflow.new
|
@@ -29,7 +30,6 @@ describe Gush::Workflow do
|
|
29
30
|
|
30
31
|
it "assigns new unique id" do
|
31
32
|
flow = TestWorkflow.new
|
32
|
-
expect(flow.id).to eq(nil)
|
33
33
|
flow.save
|
34
34
|
expect(flow.id).to_not be_nil
|
35
35
|
end
|
@@ -61,17 +61,16 @@ describe Gush::Workflow do
|
|
61
61
|
|
62
62
|
describe "#to_json" do
|
63
63
|
it "returns correct hash" do
|
64
|
-
|
65
64
|
klass = Class.new(Gush::Workflow) do
|
66
|
-
def configure
|
65
|
+
def configure(*args)
|
67
66
|
run FetchFirstJob
|
68
67
|
run PersistFirstJob, after: FetchFirstJob
|
69
68
|
end
|
70
69
|
end
|
71
70
|
|
72
|
-
result = JSON.parse(klass.
|
71
|
+
result = JSON.parse(klass.create("arg1", "arg2").to_json)
|
73
72
|
expected = {
|
74
|
-
"id"=>
|
73
|
+
"id" => an_instance_of(String),
|
75
74
|
"name" => klass.to_s,
|
76
75
|
"klass" => klass.to_s,
|
77
76
|
"status" => "pending",
|
@@ -80,67 +79,109 @@ describe Gush::Workflow do
|
|
80
79
|
"started_at" => nil,
|
81
80
|
"finished_at" => nil,
|
82
81
|
"stopped" => false,
|
82
|
+
"arguments" => ["arg1", "arg2"],
|
83
83
|
"jobs" => [
|
84
84
|
{
|
85
85
|
"name"=>"FetchFirstJob",
|
86
86
|
"klass"=>"FetchFirstJob",
|
87
|
-
"finished"=>false,
|
88
|
-
"enqueued"=>false,
|
89
|
-
"failed"=>false,
|
90
87
|
"incoming"=>[],
|
91
88
|
"outgoing"=>["PersistFirstJob"],
|
92
89
|
"finished_at"=>nil,
|
93
90
|
"started_at"=>nil,
|
94
91
|
"enqueued_at"=>nil,
|
95
92
|
"failed_at"=>nil,
|
96
|
-
"
|
93
|
+
"params" => {},
|
94
|
+
"output_payload" => nil
|
97
95
|
},
|
98
96
|
{
|
99
97
|
"name"=>"PersistFirstJob",
|
100
98
|
"klass"=>"PersistFirstJob",
|
101
|
-
"finished"=>false,
|
102
|
-
"enqueued"=>false,
|
103
|
-
"failed"=>false,
|
104
99
|
"incoming"=>["FetchFirstJob"],
|
105
100
|
"outgoing"=>[],
|
106
101
|
"finished_at"=>nil,
|
107
102
|
"started_at"=>nil,
|
108
103
|
"enqueued_at"=>nil,
|
109
104
|
"failed_at"=>nil,
|
110
|
-
"
|
105
|
+
"params" => {},
|
106
|
+
"output_payload" => nil
|
111
107
|
}
|
112
108
|
]
|
113
109
|
}
|
114
|
-
expect(result).to
|
110
|
+
expect(result).to match(expected)
|
115
111
|
end
|
116
112
|
end
|
117
113
|
|
118
114
|
describe "#find_job" do
|
119
115
|
it "finds job by its name" do
|
120
|
-
expect(TestWorkflow.
|
116
|
+
expect(TestWorkflow.create.find_job("PersistFirstJob")).to be_instance_of(PersistFirstJob)
|
121
117
|
end
|
122
118
|
end
|
123
119
|
|
124
120
|
describe "#run" do
|
121
|
+
it "allows passing additional params to the job" do
|
122
|
+
flow = Gush::Workflow.new
|
123
|
+
flow.run(Gush::Job, params: { something: 1 })
|
124
|
+
flow.save
|
125
|
+
expect(flow.jobs.first.params).to eq ({ something: 1 })
|
126
|
+
end
|
127
|
+
|
125
128
|
context "when graph is empty" do
|
126
129
|
it "adds new job with the given class as a node" do
|
127
|
-
flow = Gush::Workflow.new
|
130
|
+
flow = Gush::Workflow.new
|
128
131
|
flow.run(Gush::Job)
|
132
|
+
flow.save
|
129
133
|
expect(flow.jobs.first).to be_instance_of(Gush::Job)
|
130
134
|
end
|
131
135
|
end
|
132
136
|
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
137
|
+
it "allows `after` to accept an array of jobs" do
|
138
|
+
tree = Gush::Workflow.new
|
139
|
+
klass1 = Class.new(Gush::Job)
|
140
|
+
klass2 = Class.new(Gush::Job)
|
141
|
+
klass3 = Class.new(Gush::Job)
|
142
|
+
tree.run(klass1)
|
143
|
+
tree.run(klass2, after: [klass1, klass3])
|
144
|
+
tree.run(klass3)
|
145
|
+
|
146
|
+
tree.resolve_dependencies
|
147
|
+
|
148
|
+
expect(tree.jobs.first.outgoing).to match_array([klass2.to_s])
|
149
|
+
end
|
150
|
+
|
151
|
+
it "allows `before` to accept an array of jobs" do
|
152
|
+
tree = Gush::Workflow.new
|
153
|
+
klass1 = Class.new(Gush::Job)
|
154
|
+
klass2 = Class.new(Gush::Job)
|
155
|
+
klass3 = Class.new(Gush::Job)
|
156
|
+
tree.run(klass1)
|
157
|
+
tree.run(klass2, before: [klass1, klass3])
|
158
|
+
tree.run(klass3)
|
159
|
+
|
160
|
+
tree.resolve_dependencies
|
161
|
+
|
162
|
+
expect(tree.jobs.first.incoming).to match_array([klass2.to_s])
|
163
|
+
end
|
164
|
+
|
165
|
+
it "attaches job as a child of the job in `after` key" do
|
166
|
+
tree = Gush::Workflow.new
|
167
|
+
klass1 = Class.new(Gush::Job)
|
168
|
+
klass2 = Class.new(Gush::Job)
|
169
|
+
tree.run(klass1)
|
170
|
+
tree.run(klass2, after: klass1)
|
171
|
+
tree.resolve_dependencies
|
172
|
+
job = tree.jobs.first
|
173
|
+
expect(job.outgoing).to match_array([klass2.to_s])
|
174
|
+
end
|
175
|
+
|
176
|
+
it "attaches job as a parent of the job in `before` key" do
|
177
|
+
tree = Gush::Workflow.new
|
178
|
+
klass1 = Class.new(Gush::Job)
|
179
|
+
klass2 = Class.new(Gush::Job)
|
180
|
+
tree.run(klass1)
|
181
|
+
tree.run(klass2, before: klass1)
|
182
|
+
tree.resolve_dependencies
|
183
|
+
job = tree.jobs.first
|
184
|
+
expect(job.incoming).to match_array([klass2.to_s])
|
144
185
|
end
|
145
186
|
end
|
146
187
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: gush
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Piotrek Okoński
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-07-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: sidekiq
|
@@ -207,7 +207,7 @@ files:
|
|
207
207
|
- lib/gush/worker.rb
|
208
208
|
- lib/gush/workflow.rb
|
209
209
|
- spec/Gushfile.rb
|
210
|
-
- spec/features/
|
210
|
+
- spec/features/integration_spec.rb
|
211
211
|
- spec/lib/gush/client_spec.rb
|
212
212
|
- spec/lib/gush/configuration_spec.rb
|
213
213
|
- spec/lib/gush/job_spec.rb
|
@@ -241,7 +241,7 @@ specification_version: 4
|
|
241
241
|
summary: Fast and distributed workflow runner using only Sidekiq and Redis
|
242
242
|
test_files:
|
243
243
|
- spec/Gushfile.rb
|
244
|
-
- spec/features/
|
244
|
+
- spec/features/integration_spec.rb
|
245
245
|
- spec/lib/gush/client_spec.rb
|
246
246
|
- spec/lib/gush/configuration_spec.rb
|
247
247
|
- spec/lib/gush/job_spec.rb
|
@@ -1,31 +0,0 @@
|
|
1
|
-
require 'spec_helper'
|
2
|
-
|
3
|
-
|
4
|
-
describe "Workflows" do
|
5
|
-
it "runs the whole workflow in proper order" do
|
6
|
-
flow = TestWorkflow.create
|
7
|
-
flow.start!
|
8
|
-
|
9
|
-
expect(Gush::Worker).to have_jobs(flow.id, ["Prepare"])
|
10
|
-
|
11
|
-
Gush::Worker.perform_one
|
12
|
-
expect(Gush::Worker).to have_jobs(flow.id, ["FetchFirstJob", "FetchSecondJob"])
|
13
|
-
|
14
|
-
Gush::Worker.perform_one
|
15
|
-
expect(Gush::Worker).to have_jobs(flow.id, ["FetchSecondJob", "PersistFirstJob"])
|
16
|
-
|
17
|
-
Gush::Worker.perform_one
|
18
|
-
expect(Gush::Worker).to have_jobs(flow.id, ["PersistFirstJob", "NormalizeJob"])
|
19
|
-
|
20
|
-
Gush::Worker.perform_one
|
21
|
-
expect(Gush::Worker).to have_jobs(flow.id, ["NormalizeJob"])
|
22
|
-
|
23
|
-
Gush::Worker.perform_one
|
24
|
-
|
25
|
-
expect(Gush::Worker.jobs).to be_empty
|
26
|
-
|
27
|
-
flow = flow.reload
|
28
|
-
expect(flow).to be_finished
|
29
|
-
expect(flow).to_not be_failed
|
30
|
-
end
|
31
|
-
end
|