gush 0.1.1 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: a7d04a1298b3a3171ff5a24b5d3057e040f07ec7
4
- data.tar.gz: f63ebc5de7186f2d042e316e519807ebf1819724
3
+ metadata.gz: 5f2be71d1a76f021091ae90270ea5f01a2413ae0
4
+ data.tar.gz: c4160594dd6ad163239548623a7e67b29aa1727c
5
5
  SHA512:
6
- metadata.gz: 7492e091cce7f5c40acda1a29981db317e701a239a1e06bb9567d341fd0904ed31982a0484b1746968cdec999ce1f41aea5e2a4078756669ebc316d8352d060e
7
- data.tar.gz: 8264f120a6a876f2f265ca7270c2d2ce0177265645ed595da778433a4ff53415107cc51cb19b9f3af638e7ca4b67ec31fb6ac0969b37c214a95718a3cdd831c4
6
+ metadata.gz: 271ac326d01d2dafd4c0092e3011a5d57206b988bfc7e5e62e375b913bcb2e8750440058e07a58d60663446fc7b7254f2b91a39ed6d038976995dccc3af0952b
7
+ data.tar.gz: a28e4ef3338b305523e7d7d275aa6d3e357ddfe5161aaef95e323b12268a301c8d59352f12c68c445392df613c29ae11f38079a6838006c3ca2920c32ce418ad
data/README.md CHANGED
@@ -1,7 +1,12 @@
1
- # Gush [![Build Status](https://travis-ci.org/pokonski/gush.svg?branch=master)](https://travis-ci.org/pokonski/gush)
1
+ # Gush [![Build Status](https://travis-ci.org/chaps-io/gush.svg?branch=master)](https://travis-ci.org/chaps-io/gush)
2
+
3
+ ## [![](http://i.imgur.com/ya8Wnyl.png)](https://chaps.io) proudly made by [Chaps](https://chaps.io)
2
4
 
3
5
  Gush is a parallel workflow runner using only Redis as its message broker and Sidekiq for workers.
4
6
 
7
+ ## Theory
8
+
9
+ Gush relies on directed acyclic graphs to store dependencies, see [Parallelizing Operations With Dependencies](https://msdn.microsoft.com/en-us/magazine/dd569760.aspx) by Stephen Toub.
5
10
  ## Installation
6
11
 
7
12
  Add this line to your application's Gemfile:
@@ -26,9 +31,9 @@ Here is a complete example of a workflow you can create:
26
31
  ```ruby
27
32
  # workflows/sample_workflow.rb
28
33
  class SampleWorkflow < Gush::Workflow
29
- def configure
30
- run FetchJob1
31
- run FetchJob2
34
+ def configure(url_to_fetch_from)
35
+ run FetchJob1, params: { url: url_to_fetch_from }
36
+ run FetchJob2, params: {some_flag: true, url: 'http://url.com'}
32
37
 
33
38
  run PersistJob1, after: FetchJob1
34
39
  run PersistJob2, after: FetchJob2
@@ -52,19 +57,62 @@ For the Workflow above, the graph will look like this:
52
57
 
53
58
  ![SampleWorkflow](http://i.imgur.com/SmeRRVT.png)
54
59
 
55
- ### Defining jobs
60
+
61
+ #### Passing parameters to jobs
62
+
63
+ You can pass any primitive arguments into jobs while defining your workflow:
64
+
65
+ ```ruby
66
+ # workflows/sample_workflow.rb
67
+ class SampleWorkflow < Gush::Workflow
68
+ def configure
69
+ run FetchJob1, params: { url: "http://some.com/url" }
70
+ end
71
+ end
72
+ ```
73
+
74
+ See below to learn how to access those params inside your job.
75
+
76
+ #### Defining jobs
56
77
 
57
78
  Jobs are classes inheriting from `Gush::Job`:
58
79
 
59
80
  ```ruby
60
- #workflows/sample/fetch_job1.rb
61
81
  class FetchJob1 < Gush::Job
62
82
  def work
63
83
  # do some fetching from remote APIs
84
+
85
+ params #=> {url: "http://some.com/url"}
64
86
  end
65
87
  end
66
88
  ```
67
89
 
90
+ `params` method is a hash containing your (optional) parameters passed to `run` method in the workflow.
91
+
92
+ #### Passing arguments to workflows
93
+
94
+ Workflows can accept any primitive arguments in their constructor, which then will be availabe in your
95
+ `configure` method.
96
+
97
+ Here's an example of a workflow responsible for publishing a book:
98
+
99
+ ```ruby
100
+ # workflows/sample_workflow.rb
101
+ class PublishBookWorkflow < Gush::Workflow
102
+ def configure(url, isbn)
103
+ run FetchBook, params: { url: url }
104
+ run PublishBook, params: { book_isbn: isbn }
105
+ end
106
+ end
107
+ ```
108
+
109
+ and then create your workflow with those arguments:
110
+
111
+ ```ruby
112
+ PublishBookWorkflow.new("http://url.com/book.pdf", "978-0470081204")
113
+ ```
114
+
115
+
68
116
  ### Running workflows
69
117
 
70
118
  Now that we have defined our workflow we can use it:
@@ -72,14 +120,14 @@ Now that we have defined our workflow we can use it:
72
120
  #### 1. Initialize and save it
73
121
 
74
122
  ```ruby
75
- flow = SampleWorkflow.new
123
+ flow = SampleWorkflow.new(optional, arguments)
76
124
  flow.save # saves workflow and its jobs to Redis
77
125
  ```
78
126
 
79
127
  **or:** you can also use a shortcut:
80
128
 
81
129
  ```ruby
82
- flow = SampleWorkflow.create
130
+ flow = SampleWorkflow.create(optional, arguments)
83
131
  ```
84
132
 
85
133
  #### 2. Start workflow
@@ -99,6 +147,50 @@ flow.start!
99
147
  Now Gush will start processing jobs in background using Sidekiq
100
148
  in the order defined in `configure` method inside Workflow.
101
149
 
150
+ ### Pipelining
151
+
152
+ Gush offers a useful feature which lets you pass results of a job to its dependencies, so they can act accordingly.
153
+
154
+ **Example:**
155
+
156
+ Let's assume you have two jobs, `DownloadVideo`, `EncodeVideo`.
157
+ The latter needs to know where the first one downloaded the file to be able to open it.
158
+
159
+
160
+ ```ruby
161
+ class DownloadVideo < Gush::Job
162
+ def work
163
+ downloader = VideoDownloader.fetch("http://youtube.com/?v=someytvideo")
164
+
165
+ output(downloader.file_path)
166
+ end
167
+ end
168
+ ```
169
+
170
+ `output` method is Gush's way of saying: "I want to pass this down to my descendants".
171
+
172
+ Now, since `DownloadVideo` finished and its dependant job `EncodeVideo` started, we can access that payload down the (pipe)line:
173
+
174
+ ```ruby
175
+ class EncodeVideo < Gush::Job
176
+ def work
177
+ video_path = payloads["DownloadVideo"]
178
+ end
179
+ end
180
+ ```
181
+
182
+ `payloads` is a hash containing outputs from all parent jobs, where job class names are the keys.
183
+
184
+ **Note:** `payloads` will only contain outputs of the job's ancestors. So if job `A` depends on `B` and `C`,
185
+ the `paylods` hash will look like this:
186
+
187
+ ```ruby
188
+ {
189
+ "B" => (...),
190
+ "C" => (...)
191
+ }
192
+ ```
193
+
102
194
 
103
195
  ### Checking status:
104
196
 
@@ -126,9 +218,8 @@ flow.status
126
218
  bundle gush list
127
219
  ```
128
220
 
129
- ### Requiring workflows inside your projects
130
221
 
131
- **Skip this step if using Gush inside Rails application, workflows will already be loaded**
222
+ ### Requiring workflows inside your projects
132
223
 
133
224
  When using Gush and its CLI commands you need a Gushfile.rb in root directory.
134
225
  Gushfile should require all your Workflows and jobs, for example:
@@ -141,6 +232,11 @@ Dir[Rails.root.join("app/workflows/**/*.rb")].each do |file|
141
232
  end
142
233
  ```
143
234
 
235
+ ## Contributors
236
+
237
+ - [Mateusz Lenik](https://github.com/mlen)
238
+ - [Michał Krzyżanowski](https://github.com/krzyzak)
239
+
144
240
  ## Contributing
145
241
 
146
242
  1. Fork it ( http://github.com/pokonski/gush/fork )
data/gush.gemspec CHANGED
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |spec|
6
6
  spec.name = "gush"
7
- spec.version = "0.1.1"
7
+ spec.version = "0.1.2"
8
8
  spec.authors = ["Piotrek Okoński"]
9
9
  spec.email = ["piotrek@okonski.org"]
10
10
  spec.summary = "Fast and distributed workflow runner using only Sidekiq and Redis"
data/lib/gush/cli.rb CHANGED
@@ -131,7 +131,7 @@ module Gush
131
131
 
132
132
  def load_gushfile
133
133
  file = client.configuration.gushfile
134
- if !@gushfile.exist?
134
+ if !gushfile.exist?
135
135
  raise Thor::Error, "#{file} not found, please add it to your project".colorize(:red)
136
136
  end
137
137
 
data/lib/gush/client.rb CHANGED
@@ -117,7 +117,7 @@ module Gush
117
117
  sidekiq.push(
118
118
  'class' => Gush::Worker,
119
119
  'queue' => configuration.namespace,
120
- 'args' => [workflow_id, job.class.to_s, configuration.to_json]
120
+ 'args' => [workflow_id, job.class.to_s]
121
121
  )
122
122
  end
123
123
 
@@ -126,7 +126,7 @@ module Gush
126
126
  attr_reader :sidekiq, :redis
127
127
 
128
128
  def workflow_from_hash(hash, nodes = nil)
129
- flow = hash[:klass].constantize.new(false)
129
+ flow = hash[:klass].constantize.new
130
130
  flow.stopped = hash.fetch(:stopped, false)
131
131
  flow.id = hash[:id]
132
132
 
data/lib/gush/job.rb CHANGED
@@ -1,10 +1,8 @@
1
1
  module Gush
2
2
  class Job
3
-
4
- attr_accessor :workflow_id, :incoming, :outgoing,
5
- :finished_at, :failed_at, :started_at, :enqueued_at
6
-
7
- attr_reader :name
3
+ attr_accessor :workflow_id, :incoming, :outgoing, :params,
4
+ :finished_at, :failed_at, :started_at, :enqueued_at, :payloads
5
+ attr_reader :name, :output_payload, :params
8
6
 
9
7
  def initialize(workflow, opts = {})
10
8
  @workflow = workflow
@@ -14,18 +12,16 @@ module Gush
14
12
 
15
13
  def as_json
16
14
  {
17
- name: @name,
15
+ name: name,
18
16
  klass: self.class.to_s,
19
- finished: finished?,
20
- enqueued: enqueued?,
21
- failed: failed?,
22
- incoming: @incoming,
23
- outgoing: @outgoing,
17
+ incoming: incoming,
18
+ outgoing: outgoing,
24
19
  finished_at: finished_at,
25
20
  enqueued_at: enqueued_at,
26
21
  started_at: started_at,
27
22
  failed_at: failed_at,
28
- running: running?
23
+ params: params,
24
+ output_payload: output_payload
29
25
  }
30
26
  end
31
27
 
@@ -37,6 +33,10 @@ module Gush
37
33
  hash[:klass].constantize.new(flow, hash)
38
34
  end
39
35
 
36
+ def output(data)
37
+ @output_payload = data
38
+ end
39
+
40
40
  def work
41
41
  end
42
42
 
@@ -56,8 +56,7 @@ module Gush
56
56
  end
57
57
 
58
58
  def fail!
59
- @finished_at = current_timestamp
60
- @failed_at = current_timestamp
59
+ @finished_at = @failed_at = current_timestamp
61
60
  end
62
61
 
63
62
  def enqueued?
@@ -89,19 +88,20 @@ module Gush
89
88
  end
90
89
 
91
90
  private
92
-
93
91
  def current_timestamp
94
92
  Time.now.to_i
95
93
  end
96
94
 
97
- def assign_variables(options)
98
- @name = options[:name]
99
- @incoming = options[:incoming] || []
100
- @outgoing = options[:outgoing] || []
101
- @failed_at = options[:failed_at]
102
- @finished_at = options[:finished_at]
103
- @started_at = options[:started_at]
104
- @enqueued_at = options[:enqueued_at]
95
+ def assign_variables(opts)
96
+ @name = opts[:name]
97
+ @incoming = opts[:incoming] || []
98
+ @outgoing = opts[:outgoing] || []
99
+ @failed_at = opts[:failed_at]
100
+ @finished_at = opts[:finished_at]
101
+ @started_at = opts[:started_at]
102
+ @enqueued_at = opts[:enqueued_at]
103
+ @params = opts[:params] || {}
104
+ @output_payload = opts[:output_payload]
105
105
  end
106
106
  end
107
107
  end
data/lib/gush/worker.rb CHANGED
@@ -6,19 +6,18 @@ module Gush
6
6
  include ::Sidekiq::Worker
7
7
  sidekiq_options retry: false
8
8
 
9
- def perform(workflow_id, job_id, configuration_json)
10
- configure_client(configuration_json)
9
+ def perform(workflow_id, job_id)
10
+ setup_job(workflow_id, job_id)
11
11
 
12
- workflow = client.find_workflow(workflow_id)
13
- job = workflow.find_job(job_id)
12
+ job.payloads = incoming_payloads
14
13
 
15
14
  start = Time.now
16
- report(workflow, job, :started, start)
15
+ report(:started, start)
17
16
 
18
17
  failed = false
19
18
  error = nil
20
19
 
21
- mark_as_started(workflow, job)
20
+ mark_as_started
22
21
  begin
23
22
  job.work
24
23
  rescue Exception => e
@@ -27,46 +26,68 @@ module Gush
27
26
  end
28
27
 
29
28
  unless failed
30
- report(workflow, job, :finished, start)
31
- mark_as_finished(workflow, job)
29
+ report(:finished, start)
30
+ mark_as_finished
32
31
 
33
- enqueue_outgoing_jobs(workflow.id, job)
32
+ enqueue_outgoing_jobs
34
33
  else
35
- mark_as_failed(workflow, job)
36
- report(workflow, job, :failed, start, error.message)
34
+ mark_as_failed
35
+ report(:failed, start, error.message)
37
36
  end
38
37
  end
39
38
 
40
39
  private
40
+ attr_reader :client, :workflow, :job
41
41
 
42
- attr_reader :client
42
+ def client
43
+ @client ||= Gush::Client.new(Gush.configuration)
44
+ end
45
+
46
+ def setup_job(workflow_id, job_id)
47
+ @workflow ||= client.find_workflow(workflow_id)
48
+ @job ||= workflow.find_job(job_id)
49
+ end
50
+
51
+ def incoming_payloads
52
+ payloads = {}
53
+ job.incoming.each do |job_name|
54
+ payloads[job_name] = client.load_job(workflow.id, job_name).output_payload
55
+ end
43
56
 
44
- def configure_client(config_json)
45
- @client = Client.new(Configuration.from_json(config_json))
57
+ payloads
46
58
  end
47
59
 
48
- def mark_as_finished(workflow, job)
60
+ def mark_as_finished
49
61
  job.finish!
50
62
  client.persist_job(workflow.id, job)
51
63
  end
52
64
 
53
- def mark_as_failed(workflow, job)
65
+ def mark_as_failed
54
66
  job.fail!
55
67
  client.persist_job(workflow.id, job)
56
68
  end
57
69
 
58
- def mark_as_started(workflow, job)
70
+ def mark_as_started
59
71
  job.start!
60
72
  client.persist_job(workflow.id, job)
61
73
  end
62
74
 
63
- def report_workflow_status(workflow, job)
64
- message = {workflow_id: workflow.id, status: workflow.status, started_at: workflow.started_at, finished_at: workflow.finished_at }
65
- client.workflow_report(message)
75
+ def report_workflow_status
76
+ client.workflow_report({
77
+ workflow_id: workflow.id,
78
+ status: workflow.status,
79
+ started_at: workflow.started_at,
80
+ finished_at: workflow.finished_at
81
+ })
66
82
  end
67
83
 
68
- def report(workflow, job, status, start, error = nil)
69
- message = {status: status, workflow_id: workflow.id, job: job.name, duration: elapsed(start)}
84
+ def report(status, start, error = nil)
85
+ message = {
86
+ status: status,
87
+ workflow_id: workflow.id,
88
+ job: job.name,
89
+ duration: elapsed(start)
90
+ }
70
91
  message[:error] = error if error
71
92
  client.worker_report(message)
72
93
  end
@@ -75,11 +96,11 @@ module Gush
75
96
  (Time.now - start).to_f.round(3)
76
97
  end
77
98
 
78
- def enqueue_outgoing_jobs(workflow_id, job)
99
+ def enqueue_outgoing_jobs
79
100
  job.outgoing.each do |job_name|
80
- out = client.load_job(workflow_id, job_name)
101
+ out = client.load_job(workflow.id, job_name)
81
102
  if out.ready_to_start?
82
- client.enqueue_job(workflow_id, out)
103
+ client.enqueue_job(workflow.id, out)
83
104
  end
84
105
  end
85
106
  end
data/lib/gush/workflow.rb CHANGED
@@ -2,19 +2,15 @@ require 'securerandom'
2
2
 
3
3
  module Gush
4
4
  class Workflow
5
- attr_accessor :id, :jobs, :stopped, :persisted
5
+ attr_accessor :id, :jobs, :stopped, :persisted, :arguments
6
6
 
7
- def initialize(should_run_configure = true)
7
+ def initialize(*args)
8
8
  @id = id
9
9
  @jobs = []
10
10
  @dependencies = []
11
11
  @persisted = false
12
12
  @stopped = false
13
-
14
- if should_run_configure
15
- configure
16
- create_dependencies
17
- end
13
+ @arguments = args
18
14
  end
19
15
 
20
16
  def self.find(id)
@@ -28,14 +24,12 @@ module Gush
28
24
  end
29
25
 
30
26
  def save
31
- if @id.nil?
32
- assign_id
33
- end
34
-
27
+ configure(*@arguments)
28
+ resolve_dependencies
35
29
  client.persist_workflow(self)
36
30
  end
37
31
 
38
- def configure
32
+ def configure(*args)
39
33
  end
40
34
 
41
35
  def mark_as_stopped
@@ -58,7 +52,7 @@ module Gush
58
52
  @stopped = false
59
53
  end
60
54
 
61
- def create_dependencies
55
+ def resolve_dependencies
62
56
  @dependencies.each do |dependency|
63
57
  from = find_job(dependency[:from])
64
58
  to = find_job(dependency[:to])
@@ -69,7 +63,7 @@ module Gush
69
63
  end
70
64
 
71
65
  def find_job(name)
72
- @jobs.find { |node| node.name == name.to_s || node.class.to_s == name.to_s }
66
+ jobs.find { |node| node.name == name.to_s || node.class.to_s == name.to_s }
73
67
  end
74
68
 
75
69
  def finished?
@@ -88,23 +82,29 @@ module Gush
88
82
  stopped
89
83
  end
90
84
 
91
- def run(klass, deps = {})
92
- node = klass.new(self, name: klass.to_s)
93
- @jobs << node
85
+ def run(klass, opts = {})
86
+ options =
87
+
88
+ node = klass.new(self, {
89
+ name: klass.to_s,
90
+ params: opts.fetch(:params, {})
91
+ })
94
92
 
95
- deps_after = [*deps[:after]]
93
+ jobs << node
94
+
95
+ deps_after = [*opts[:after]]
96
96
  deps_after.each do |dep|
97
97
  @dependencies << {from: dep.to_s, to: klass.to_s }
98
98
  end
99
99
 
100
- deps_before = [*deps[:before]]
100
+ deps_before = [*opts[:before]]
101
101
  deps_before.each do |dep|
102
102
  @dependencies << {from: klass.to_s, to: dep.to_s }
103
103
  end
104
104
  end
105
105
 
106
106
  def reload
107
- self.class.find(@id)
107
+ self.class.find(id)
108
108
  end
109
109
 
110
110
  def initial_jobs
@@ -138,11 +138,12 @@ module Gush
138
138
  name = self.class.to_s
139
139
  {
140
140
  name: name,
141
- id: @id,
142
- total: @jobs.count,
143
- finished: @jobs.count(&:finished?),
141
+ id: id,
142
+ arguments: @arguments,
143
+ total: jobs.count,
144
+ finished: jobs.count(&:finished?),
144
145
  klass: name,
145
- jobs: @jobs.map(&:as_json),
146
+ jobs: jobs.map(&:as_json),
146
147
  status: status,
147
148
  stopped: stopped,
148
149
  started_at: started_at,
@@ -158,12 +159,12 @@ module Gush
158
159
  ObjectSpace.each_object(Class).select { |klass| klass < self }
159
160
  end
160
161
 
161
- private
162
-
163
- def assign_id
164
- @id = client.next_free_id
162
+ def id
163
+ @id ||= client.next_free_id
165
164
  end
166
165
 
166
+ private
167
+
167
168
  def client
168
169
  @client ||= Client.new
169
170
  end
@@ -0,0 +1,72 @@
1
+ require 'spec_helper'
2
+
3
+
4
+ describe "Workflows" do
5
+ it "runs the whole workflow in proper order" do
6
+ flow = TestWorkflow.create
7
+ flow.start!
8
+
9
+ expect(Gush::Worker).to have_jobs(flow.id, ["Prepare"])
10
+
11
+ Gush::Worker.perform_one
12
+ expect(Gush::Worker).to have_jobs(flow.id, ["FetchFirstJob", "FetchSecondJob"])
13
+
14
+ Gush::Worker.perform_one
15
+ expect(Gush::Worker).to have_jobs(flow.id, ["FetchSecondJob", "PersistFirstJob"])
16
+
17
+ Gush::Worker.perform_one
18
+ expect(Gush::Worker).to have_jobs(flow.id, ["PersistFirstJob", "NormalizeJob"])
19
+
20
+ Gush::Worker.perform_one
21
+ expect(Gush::Worker).to have_jobs(flow.id, ["NormalizeJob"])
22
+
23
+ Gush::Worker.perform_one
24
+
25
+ expect(Gush::Worker.jobs).to be_empty
26
+
27
+ flow = flow.reload
28
+ expect(flow).to be_finished
29
+ expect(flow).to_not be_failed
30
+ end
31
+
32
+ it "passes payloads down the workflow" do
33
+ class UpcaseJob < Gush::Job
34
+ def work
35
+ output params[:input].upcase
36
+ end
37
+ end
38
+
39
+ class PrefixJob < Gush::Job
40
+ def work
41
+ output params[:prefix].capitalize
42
+ end
43
+ end
44
+
45
+ class PrependJob < Gush::Job
46
+ def work
47
+ string = "#{payloads["PrefixJob"]}: #{payloads["UpcaseJob"]}"
48
+ output string
49
+ end
50
+ end
51
+
52
+ class PayloadWorkflow < Gush::Workflow
53
+ def configure
54
+ run UpcaseJob, params: {input: "some text"}
55
+ run PrefixJob, params: {prefix: "a prefix"}
56
+ run PrependJob, after: [UpcaseJob, PrefixJob]
57
+ end
58
+ end
59
+
60
+ flow = PayloadWorkflow.create
61
+ flow.start!
62
+
63
+ Gush::Worker.perform_one
64
+ expect(flow.reload.find_job("UpcaseJob").output_payload).to eq("SOME TEXT")
65
+
66
+ Gush::Worker.perform_one
67
+ expect(flow.reload.find_job("PrefixJob").output_payload).to eq("A prefix")
68
+
69
+ Gush::Worker.perform_one
70
+ expect(flow.reload.find_job("PrependJob").output_payload).to eq("A prefix: SOME TEXT")
71
+ end
72
+ end
@@ -16,8 +16,7 @@ describe Gush::Client do
16
16
 
17
17
  context "when given workflow exists" do
18
18
  it "returns Workflow object" do
19
- expected_workflow = TestWorkflow.new(SecureRandom.uuid)
20
- client.persist_workflow(expected_workflow)
19
+ expected_workflow = TestWorkflow.create
21
20
  workflow = client.find_workflow(expected_workflow.id)
22
21
 
23
22
  expect(workflow.id).to eq(expected_workflow.id)
@@ -28,8 +27,7 @@ describe Gush::Client do
28
27
 
29
28
  describe "#start_workflow" do
30
29
  it "enqueues next jobs from the workflow" do
31
- workflow = TestWorkflow.new
32
- client.persist_workflow(workflow)
30
+ workflow = TestWorkflow.create
33
31
  expect {
34
32
  client.start_workflow(workflow)
35
33
  }.to change{Gush::Worker.jobs.count}.from(0).to(1)
@@ -38,15 +36,14 @@ describe Gush::Client do
38
36
  it "removes stopped flag when the workflow is started" do
39
37
  workflow = TestWorkflow.new
40
38
  workflow.mark_as_stopped
41
- client.persist_workflow(workflow)
39
+ workflow.save
42
40
  expect {
43
41
  client.start_workflow(workflow)
44
42
  }.to change{client.find_workflow(workflow.id).stopped?}.from(true).to(false)
45
43
  end
46
44
 
47
45
  it "marks the enqueued jobs as enqueued" do
48
- workflow = TestWorkflow.new
49
- client.persist_workflow(workflow)
46
+ workflow = TestWorkflow.create
50
47
  client.start_workflow(workflow)
51
48
  job = workflow.reload.find_job("Prepare")
52
49
  expect(job.enqueued?).to eq(true)
@@ -55,8 +52,7 @@ describe Gush::Client do
55
52
 
56
53
  describe "#stop_workflow" do
57
54
  it "marks the workflow as stopped" do
58
- workflow = TestWorkflow.new
59
- client.persist_workflow(workflow)
55
+ workflow = TestWorkflow.create
60
56
  expect {
61
57
  client.stop_workflow(workflow.id)
62
58
  }.to change{client.find_workflow(workflow.id).stopped?}.from(false).to(true)
@@ -76,8 +72,7 @@ describe Gush::Client do
76
72
 
77
73
  describe "#destroy_workflow" do
78
74
  it "removes all Redis keys related to the workflow" do
79
- workflow = TestWorkflow.new
80
- client.persist_workflow(workflow)
75
+ workflow = TestWorkflow.create
81
76
  expect(redis.keys("gush.workflows.#{workflow.id}").length).to eq(1)
82
77
  expect(redis.keys("gush.jobs.#{workflow.id}.*").length).to eq(5)
83
78
 
@@ -98,16 +93,14 @@ describe Gush::Client do
98
93
 
99
94
  describe "#all_workflows" do
100
95
  it "returns all registered workflows" do
101
- workflow = TestWorkflow.new(SecureRandom.uuid)
102
- client.persist_workflow(workflow)
96
+ workflow = TestWorkflow.create
103
97
  workflows = client.all_workflows
104
98
  expect(workflows.map(&:id)).to eq([workflow.id])
105
99
  end
106
100
  end
107
101
 
108
102
  it "should be able to handle outdated data format" do
109
- workflow = TestWorkflow.new
110
- client.persist_workflow(workflow)
103
+ workflow = TestWorkflow.create
111
104
 
112
105
  # malform the data
113
106
  hash = Gush::JSON.decode(redis.get("gush.workflows.#{workflow.id}"), symbolize_keys: true)
@@ -2,6 +2,13 @@ require 'spec_helper'
2
2
 
3
3
  describe Gush::Job do
4
4
 
5
+ describe "#output" do
6
+ it "saves output to output_payload" do
7
+ job = described_class.new(name: "a-job")
8
+ job.output "something"
9
+ expect(job.output_payload).to eq("something")
10
+ end
11
+ end
5
12
  describe "#fail!" do
6
13
  it "sets finished and failed to true and records time" do
7
14
  job = described_class.new(name: "a-job")
@@ -59,16 +66,14 @@ describe Gush::Job do
59
66
  expected = {
60
67
  name: "a-job",
61
68
  klass: "Gush::Job",
62
- finished: true,
63
- enqueued: true,
64
- failed: false,
65
69
  incoming: [],
66
70
  outgoing: [],
67
71
  failed_at: nil,
68
72
  started_at: nil,
69
73
  finished_at: 123,
70
74
  enqueued_at: 120,
71
- running: false
75
+ params: {},
76
+ output_payload: nil
72
77
  }
73
78
  expect(job.as_json).to eq(expected)
74
79
  end
@@ -82,9 +87,6 @@ describe Gush::Job do
82
87
  {
83
88
  klass: 'Gush::Job',
84
89
  name: 'gob',
85
- finished: true,
86
- failed: true,
87
- enqueued: true,
88
90
  incoming: ['a', 'b'],
89
91
  outgoing: ['c'],
90
92
  failed_at: 123,
@@ -23,7 +23,7 @@ describe Gush::Worker do
23
23
  allow(job).to receive(:work).and_raise(StandardError)
24
24
  expect(client).to receive(:worker_report).with(hash_including(status: :failed)).ordered
25
25
 
26
- subject.perform(workflow.id, "Prepare", config)
26
+ subject.perform(workflow.id, "Prepare")
27
27
  expect(workflow.find_job("Prepare")).to be_failed
28
28
  end
29
29
 
@@ -31,7 +31,7 @@ describe Gush::Worker do
31
31
  allow(job).to receive(:work).and_raise(StandardError)
32
32
  expect(client).to receive(:worker_report).with(hash_including(status: :failed)).ordered
33
33
 
34
- subject.perform(workflow.id, "Prepare", config)
34
+ subject.perform(workflow.id, "Prepare")
35
35
  end
36
36
  end
37
37
 
@@ -40,13 +40,13 @@ describe Gush::Worker do
40
40
  expect(subject).to receive(:mark_as_finished)
41
41
  expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
42
42
 
43
- subject.perform(workflow.id, "Prepare", config)
43
+ subject.perform(workflow.id, "Prepare")
44
44
  end
45
45
 
46
46
  it "reports that job succedeed" do
47
47
  expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
48
48
 
49
- subject.perform(workflow.id, "Prepare", config)
49
+ subject.perform(workflow.id, "Prepare")
50
50
  end
51
51
  end
52
52
 
@@ -54,14 +54,14 @@ describe Gush::Worker do
54
54
  expect(job).to receive(:work)
55
55
  expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
56
56
 
57
- subject.perform(workflow.id, "Prepare", config)
57
+ subject.perform(workflow.id, "Prepare")
58
58
  end
59
59
 
60
60
  it "reports when the job is started" do
61
61
  allow(client).to receive(:worker_report)
62
62
  expect(client).to receive(:worker_report).with(hash_including(status: :finished)).ordered
63
63
 
64
- subject.perform(workflow.id, "Prepare", config)
64
+ subject.perform(workflow.id, "Prepare")
65
65
  end
66
66
  end
67
67
  end
@@ -1,25 +1,26 @@
1
1
  require 'spec_helper'
2
2
 
3
3
  describe Gush::Workflow do
4
- subject { TestWorkflow.new("test-workflow") }
4
+ subject { TestWorkflow.create }
5
5
 
6
6
  describe "#initialize" do
7
- context "when configure option is true" do
8
- it "runs #configure method " do
9
- expect_any_instance_of(TestWorkflow).to receive(:configure)
10
- TestWorkflow.new(true)
11
- end
12
- end
7
+ end
13
8
 
14
- context "when configure option is false" do
15
- it "it doesn't run #configure method " do
16
- expect_any_instance_of(TestWorkflow).to_not receive(:configure)
17
- TestWorkflow.new(false)
9
+ describe "#save" do
10
+ it "passes constructor arguments to the method" do
11
+ klass = Class.new(Gush::Workflow) do
12
+ def configure(*args)
13
+ run FetchFirstJob
14
+ run PersistFirstJob, after: FetchFirstJob
15
+ end
18
16
  end
17
+
18
+ flow = klass.new("arg1", "arg2")
19
+
20
+ expect(flow).to receive(:configure).with("arg1", "arg2")
21
+ flow.save
19
22
  end
20
- end
21
23
 
22
- describe "#save" do
23
24
  context "workflow not persisted" do
24
25
  it "sets persisted to true" do
25
26
  flow = TestWorkflow.new
@@ -29,7 +30,6 @@ describe Gush::Workflow do
29
30
 
30
31
  it "assigns new unique id" do
31
32
  flow = TestWorkflow.new
32
- expect(flow.id).to eq(nil)
33
33
  flow.save
34
34
  expect(flow.id).to_not be_nil
35
35
  end
@@ -61,17 +61,16 @@ describe Gush::Workflow do
61
61
 
62
62
  describe "#to_json" do
63
63
  it "returns correct hash" do
64
-
65
64
  klass = Class.new(Gush::Workflow) do
66
- def configure
65
+ def configure(*args)
67
66
  run FetchFirstJob
68
67
  run PersistFirstJob, after: FetchFirstJob
69
68
  end
70
69
  end
71
70
 
72
- result = JSON.parse(klass.new("workflow").to_json)
71
+ result = JSON.parse(klass.create("arg1", "arg2").to_json)
73
72
  expected = {
74
- "id"=>nil,
73
+ "id" => an_instance_of(String),
75
74
  "name" => klass.to_s,
76
75
  "klass" => klass.to_s,
77
76
  "status" => "pending",
@@ -80,67 +79,109 @@ describe Gush::Workflow do
80
79
  "started_at" => nil,
81
80
  "finished_at" => nil,
82
81
  "stopped" => false,
82
+ "arguments" => ["arg1", "arg2"],
83
83
  "jobs" => [
84
84
  {
85
85
  "name"=>"FetchFirstJob",
86
86
  "klass"=>"FetchFirstJob",
87
- "finished"=>false,
88
- "enqueued"=>false,
89
- "failed"=>false,
90
87
  "incoming"=>[],
91
88
  "outgoing"=>["PersistFirstJob"],
92
89
  "finished_at"=>nil,
93
90
  "started_at"=>nil,
94
91
  "enqueued_at"=>nil,
95
92
  "failed_at"=>nil,
96
- "running" => false
93
+ "params" => {},
94
+ "output_payload" => nil
97
95
  },
98
96
  {
99
97
  "name"=>"PersistFirstJob",
100
98
  "klass"=>"PersistFirstJob",
101
- "finished"=>false,
102
- "enqueued"=>false,
103
- "failed"=>false,
104
99
  "incoming"=>["FetchFirstJob"],
105
100
  "outgoing"=>[],
106
101
  "finished_at"=>nil,
107
102
  "started_at"=>nil,
108
103
  "enqueued_at"=>nil,
109
104
  "failed_at"=>nil,
110
- "running" => false
105
+ "params" => {},
106
+ "output_payload" => nil
111
107
  }
112
108
  ]
113
109
  }
114
- expect(result).to eq(expected)
110
+ expect(result).to match(expected)
115
111
  end
116
112
  end
117
113
 
118
114
  describe "#find_job" do
119
115
  it "finds job by its name" do
120
- expect(TestWorkflow.new("test").find_job("PersistFirstJob")).to be_instance_of(PersistFirstJob)
116
+ expect(TestWorkflow.create.find_job("PersistFirstJob")).to be_instance_of(PersistFirstJob)
121
117
  end
122
118
  end
123
119
 
124
120
  describe "#run" do
121
+ it "allows passing additional params to the job" do
122
+ flow = Gush::Workflow.new
123
+ flow.run(Gush::Job, params: { something: 1 })
124
+ flow.save
125
+ expect(flow.jobs.first.params).to eq ({ something: 1 })
126
+ end
127
+
125
128
  context "when graph is empty" do
126
129
  it "adds new job with the given class as a node" do
127
- flow = Gush::Workflow.new("workflow")
130
+ flow = Gush::Workflow.new
128
131
  flow.run(Gush::Job)
132
+ flow.save
129
133
  expect(flow.jobs.first).to be_instance_of(Gush::Job)
130
134
  end
131
135
  end
132
136
 
133
- context "when last node is a job" do
134
- it "attaches job as a child of the last inserted job" do
135
- tree = Gush::Workflow.new("workflow")
136
- klass1 = Class.new(Gush::Job)
137
- klass2 = Class.new(Gush::Job)
138
- tree.run(klass1)
139
- tree.run(klass2, after: klass1)
140
- tree.create_dependencies
141
- expect(tree.jobs.first).to be_an_instance_of(klass1)
142
- expect(tree.jobs.first.outgoing.first).to eq(klass2.to_s)
143
- end
137
+ it "allows `after` to accept an array of jobs" do
138
+ tree = Gush::Workflow.new
139
+ klass1 = Class.new(Gush::Job)
140
+ klass2 = Class.new(Gush::Job)
141
+ klass3 = Class.new(Gush::Job)
142
+ tree.run(klass1)
143
+ tree.run(klass2, after: [klass1, klass3])
144
+ tree.run(klass3)
145
+
146
+ tree.resolve_dependencies
147
+
148
+ expect(tree.jobs.first.outgoing).to match_array([klass2.to_s])
149
+ end
150
+
151
+ it "allows `before` to accept an array of jobs" do
152
+ tree = Gush::Workflow.new
153
+ klass1 = Class.new(Gush::Job)
154
+ klass2 = Class.new(Gush::Job)
155
+ klass3 = Class.new(Gush::Job)
156
+ tree.run(klass1)
157
+ tree.run(klass2, before: [klass1, klass3])
158
+ tree.run(klass3)
159
+
160
+ tree.resolve_dependencies
161
+
162
+ expect(tree.jobs.first.incoming).to match_array([klass2.to_s])
163
+ end
164
+
165
+ it "attaches job as a child of the job in `after` key" do
166
+ tree = Gush::Workflow.new
167
+ klass1 = Class.new(Gush::Job)
168
+ klass2 = Class.new(Gush::Job)
169
+ tree.run(klass1)
170
+ tree.run(klass2, after: klass1)
171
+ tree.resolve_dependencies
172
+ job = tree.jobs.first
173
+ expect(job.outgoing).to match_array([klass2.to_s])
174
+ end
175
+
176
+ it "attaches job as a parent of the job in `before` key" do
177
+ tree = Gush::Workflow.new
178
+ klass1 = Class.new(Gush::Job)
179
+ klass2 = Class.new(Gush::Job)
180
+ tree.run(klass1)
181
+ tree.run(klass2, before: klass1)
182
+ tree.resolve_dependencies
183
+ job = tree.jobs.first
184
+ expect(job.incoming).to match_array([klass2.to_s])
144
185
  end
145
186
  end
146
187
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gush
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.1
4
+ version: 0.1.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Piotrek Okoński
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-05-16 00:00:00.000000000 Z
11
+ date: 2015-07-22 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: sidekiq
@@ -207,7 +207,7 @@ files:
207
207
  - lib/gush/worker.rb
208
208
  - lib/gush/workflow.rb
209
209
  - spec/Gushfile.rb
210
- - spec/features/workflows_spec.rb
210
+ - spec/features/integration_spec.rb
211
211
  - spec/lib/gush/client_spec.rb
212
212
  - spec/lib/gush/configuration_spec.rb
213
213
  - spec/lib/gush/job_spec.rb
@@ -241,7 +241,7 @@ specification_version: 4
241
241
  summary: Fast and distributed workflow runner using only Sidekiq and Redis
242
242
  test_files:
243
243
  - spec/Gushfile.rb
244
- - spec/features/workflows_spec.rb
244
+ - spec/features/integration_spec.rb
245
245
  - spec/lib/gush/client_spec.rb
246
246
  - spec/lib/gush/configuration_spec.rb
247
247
  - spec/lib/gush/job_spec.rb
@@ -1,31 +0,0 @@
1
- require 'spec_helper'
2
-
3
-
4
- describe "Workflows" do
5
- it "runs the whole workflow in proper order" do
6
- flow = TestWorkflow.create
7
- flow.start!
8
-
9
- expect(Gush::Worker).to have_jobs(flow.id, ["Prepare"])
10
-
11
- Gush::Worker.perform_one
12
- expect(Gush::Worker).to have_jobs(flow.id, ["FetchFirstJob", "FetchSecondJob"])
13
-
14
- Gush::Worker.perform_one
15
- expect(Gush::Worker).to have_jobs(flow.id, ["FetchSecondJob", "PersistFirstJob"])
16
-
17
- Gush::Worker.perform_one
18
- expect(Gush::Worker).to have_jobs(flow.id, ["PersistFirstJob", "NormalizeJob"])
19
-
20
- Gush::Worker.perform_one
21
- expect(Gush::Worker).to have_jobs(flow.id, ["NormalizeJob"])
22
-
23
- Gush::Worker.perform_one
24
-
25
- expect(Gush::Worker.jobs).to be_empty
26
-
27
- flow = flow.reload
28
- expect(flow).to be_finished
29
- expect(flow).to_not be_failed
30
- end
31
- end