jongleur 1.0.4 → 1.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -1
- data/README.md +68 -36
- data/examples/example_tasks.rb +49 -0
- data/examples/simple.rb +25 -0
- data/jongleur.gemspec +3 -1
- data/lib/jongleur/api.rb +9 -7
- data/lib/jongleur/implementation.rb +7 -1
- data/lib/jongleur/version.rb +1 -1
- data/lib/jongleur.rb +6 -1
- metadata +20 -6
- data/exe/test_gem.sh +0 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1db9f251078cde21dde04e7c3c1c39c9ad6507ac
|
4
|
+
data.tar.gz: '0609f7301f92b48d44f9c7f62d5a5456b927945e'
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1a849e5637bd3aa31f3cc9f80ebf335ee9691f2d54033bde1ace51b05fa86a4f78abb995ef647e1cb025f0050697bef151421d0530db3c1486d8baaf23df5766
|
7
|
+
data.tar.gz: eb9d2f5f43b0fa02ff67620f1b4731de57491d70e437e80c9f0f58153d765d7118ba0a6280022846ec903cb02b1c43431ddf1c0b5ddd6f92c8e1d51f0d0c12f6
|
data/CHANGELOG.md
CHANGED
@@ -16,4 +16,12 @@ All notable changes to this project will be documented in this file.
|
|
16
16
|
|
17
17
|
## [1.0.4] - 20-Oct-2018
|
18
18
|
### Added
|
19
|
-
- bin/setup enhanced to install Graphviz dependency
|
19
|
+
- bin/setup enhanced to install Graphviz dependency
|
20
|
+
|
21
|
+
## [1.1.0] - 31-Dec-2018
|
22
|
+
### Added
|
23
|
+
- 'Completed' asynchronous callback when execution is finished (via Hollerback gem)
|
24
|
+
- Added finish_time atrribute to task_matrix as to timestamp process completion
|
25
|
+
|
26
|
+
### Changed
|
27
|
+
- Re-written specs to use async callback
|
data/README.md
CHANGED
@@ -4,7 +4,7 @@
|
|
4
4
|
|
5
5
|
Jongleur is a process scheduler and manager. It allows its users to declare a number of executable tasks as Ruby classes, define precedence between those tasks and run each task as a separate process.
|
6
6
|
|
7
|
-
Jongleur is particularly useful for implementing workflows
|
7
|
+
Jongleur is particularly useful for implementing workflows modelled as a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
|
8
8
|
(Directed Acyclic Graph), but can be also used to run multiple tasks in parallel or even sequential workflows where each task needs to run as a separate OS process.
|
9
9
|
|
10
10
|
## Environment
|
@@ -14,6 +14,16 @@ It will work on Linux and Mac OS but not on Windows.
|
|
14
14
|
|
15
15
|
Jongleur has been tested with MRuby 2.4.3, 2.4.4, 2.5.0 and 2.5.1. I would also expect it to work with other Ruby implementations too, such as JRuby or Rubinius though it hasn't yet been tested on those.
|
16
16
|
|
17
|
+
## Pre-requisites
|
18
|
+
|
19
|
+
This gem depends on the [Graphviz](https://www.graphviz.org/) package for drawing graphs. If this isn't already installed on your system please install with
|
20
|
+
|
21
|
+
`$ sudo apt-get install graphviz` (Linux)
|
22
|
+
|
23
|
+
or
|
24
|
+
|
25
|
+
`$ brew cask install graphviz`(Mac OS)
|
26
|
+
|
17
27
|
## Installation
|
18
28
|
|
19
29
|
Add this line to your application's Gemfile:
|
@@ -31,9 +41,11 @@ Or install it yourself as:
|
|
31
41
|
$ gem install jongleur
|
32
42
|
|
33
43
|
|
44
|
+
In either case, call `require jongleur` before using the gem.
|
45
|
+
|
34
46
|
## What does it do?
|
35
47
|
|
36
|
-
In a nutshell, Jongleur keeps track of a number of tasks and executes them as separate OS processes according to their precedence criteria. For instance, if there are 3 tasks A, B and C, and task C depends on A and B, Jongleur will start executing A and B in separate processes (i.e
|
48
|
+
In a nutshell, Jongleur keeps track of a number of tasks and executes them as separate OS processes according to their precedence criteria. For instance, if there are 3 tasks A, B and C, and task C depends on A and B, Jongleur will start executing A and B in separate processes (i.e.. in parallel) and will wait until they are both finished before it executes C in a separate process.
|
37
49
|
|
38
50
|
Jongleur is ideal for running workflows represented as DAGs, but is also useful for simply running tasks in parallel or for whenever you need some multi-processing capability.
|
39
51
|
|
@@ -68,9 +80,9 @@ run only after this Task is finished. So in the above example:
|
|
68
80
|
|
69
81
|
* Tasks Q, R and T can only start after task S has finished.
|
70
82
|
* Task R can only start after Q has finished.
|
71
|
-
* Tasks T and T have no
|
83
|
+
* Tasks T and T have no dependants. No other task need wait for them.
|
72
84
|
|
73
|
-
__N.B:__ Since the _Task Graph_ is a Hash, any duplicate key entries will be
|
85
|
+
__N.B:__ Since the _Task Graph_ is a Hash, any duplicate key entries will be overridden. For instance, if this Task Graph
|
74
86
|
|
75
87
|
```
|
76
88
|
my_task_graph = { A: [:B, :C], B: [:D] }
|
@@ -97,20 +109,21 @@ Jongleur::API.task_matrix
|
|
97
109
|
After defining your Task Graph and before running Jongleur, your _Task Matrix_ should look like this:
|
98
110
|
|
99
111
|
```
|
100
|
-
#<Jongleur::Task name=:A, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
101
|
-
#<Jongleur::Task name=:B, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
102
|
-
#<Jongleur::Task name=:C, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
103
|
-
#<Jongleur::Task name=:D, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
104
|
-
#<Jongleur::Task name=:E, pid=-1, running=false, exit_status=nil, success_status=nil>
|
112
|
+
#<Jongleur::Task name=:A, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
113
|
+
#<Jongleur::Task name=:B, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
114
|
+
#<Jongleur::Task name=:C, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
115
|
+
#<Jongleur::Task name=:D, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
116
|
+
#<Jongleur::Task name=:E, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>
|
117
|
+
|
105
118
|
```
|
106
119
|
After Jongleur finishes, your _Task Matrix_ will look something like this:
|
107
120
|
|
108
121
|
```
|
109
|
-
#<Jongleur::Task name=:A, pid=95117, running=false, exit_status=0, success_status=true>
|
110
|
-
#<Jongleur::Task name=:B, pid=95118, running=false, exit_status=0, success_status=true>
|
111
|
-
#<Jongleur::Task name=:C, pid=95120, running=false, exit_status=0, success_status=true>
|
112
|
-
#<Jongleur::Task name=:D, pid=95122, running=false, exit_status=0, success_status=true>
|
113
|
-
#<Jongleur::Task name=:E, pid=95123, running=false, exit_status=0, success_status=true>
|
122
|
+
#<Jongleur::Task name=:A, pid=95117, running=false, exit_status=0, finish_time=1546279899.005019, success_status=true>
|
123
|
+
#<Jongleur::Task name=:B, pid=95118, running=false, exit_status=0, finish_time=1546279901.0053551, success_status=true>
|
124
|
+
#<Jongleur::Task name=:C, pid=95120, running=false, exit_status=0, finish_time=1546279900.0213592, success_status=true>
|
125
|
+
#<Jongleur::Task name=:D, pid=95122, running=false, exit_status=0, finish_time=1546279912.0673511, success_status=true>
|
126
|
+
#<Jongleur::Task name=:E, pid=95123, running=false, exit_status=0, finish_time=1546279909.0876418, success_status=true>
|
114
127
|
```
|
115
128
|
|
116
129
|
The `Jongleur::Task` attribute values are as follows
|
@@ -119,6 +132,7 @@ The `Jongleur::Task` attribute values are as follows
|
|
119
132
|
* pid : the Task process id (`nil` if the task hasn't yet ran)
|
120
133
|
* running : `true` if task is currently running
|
121
134
|
* exit_status : usually 0 if process finished without errors, <>0 or `nil` otherwise
|
135
|
+
* finish_time : the Task's completion timestamp as a flating point number of seconds since Epoch
|
122
136
|
* success_status : `true` if process finished successfully, `false` if it didn't or `nil` if process didn't exit at all
|
123
137
|
|
124
138
|
|
@@ -131,11 +145,10 @@ This is the implementation template for a Task. For each Task in your Task Graph
|
|
131
145
|
|
132
146
|
Using Jongleur is easy:
|
133
147
|
|
134
|
-
1. (Optional)
|
148
|
+
1. (Optional) Add `include jongleur` so that you won't have to namespace every api call.
|
135
149
|
|
136
150
|
2. Define your Task Graph
|
137
151
|
|
138
|
-
|
139
152
|
test_graph = {
|
140
153
|
A: [:B, :C],
|
141
154
|
B: [:D],
|
@@ -144,17 +157,17 @@ Using Jongleur is easy:
|
|
144
157
|
E: []
|
145
158
|
}
|
146
159
|
|
160
|
+
Each Task corresponds to a Ruby class with an `execute` method
|
147
161
|
|
148
162
|
3. Add your Task Graph to Jongleur
|
149
163
|
|
150
|
-
|
151
164
|
API.add_task_graph test_graph
|
152
165
|
|
153
|
-
=> [#<struct Jongleur::Task name=:A, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
154
|
-
#<struct Jongleur::Task name=:B, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
155
|
-
#<struct Jongleur::Task name=:C, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
156
|
-
#<struct Jongleur::Task name=:D, pid=-1, running=false, exit_status=nil, success_status=nil>,
|
157
|
-
#<struct Jongleur::Task name=:E, pid=-1, running=false, exit_status=nil, success_status=nil>]
|
166
|
+
=> [#<struct Jongleur::Task name=:A, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
167
|
+
#<struct Jongleur::Task name=:B, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
168
|
+
#<struct Jongleur::Task name=:C, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
169
|
+
#<struct Jongleur::Task name=:D, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
|
170
|
+
#<struct Jongleur::Task name=:E, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>]
|
158
171
|
|
159
172
|
|
160
173
|
Jongleur will show you the Task Matrix for your Task Graph with all attributes set at their initial values, obviously, since the Tasks haven't ran yet.
|
@@ -170,22 +183,29 @@ Using Jongleur is easy:
|
|
170
183
|
|
171
184
|
<img src="./bin/img/DAG_graph_1.png" width="225" height="450" alt="ETL DAG">
|
172
185
|
|
173
|
-
5. Implement your tasks. To do that you have to
|
186
|
+
5. Implement your tasks. To do that you have to
|
187
|
+
|
188
|
+
* create a new class, based on `WorkerTask` and named as in your Task Graph
|
189
|
+
* define an `#execute` method in your class. This is the method that Jongleur will call to run the Task.
|
190
|
+
|
191
|
+
For instance task A from your Task Graph may look something like that:
|
174
192
|
|
175
193
|
|
176
194
|
class A < Jongleur::WorkerTask
|
177
195
|
@desc = 'this is task A'
|
178
196
|
def execute
|
179
|
-
sleep 1
|
197
|
+
sleep 1 # do something
|
180
198
|
'A is running... '
|
181
199
|
end
|
182
200
|
end
|
183
201
|
|
184
|
-
You'll have to do the same for Tasks B, C, D and E, as these
|
202
|
+
You'll have to do the same for Tasks B, C, D and E, as these are the tasks declared in the Task Graph.
|
185
203
|
|
186
|
-
6. Run the tasks
|
204
|
+
6. Run the tasks and implement the `completed` callback. This will be called asynchronously when Jongleur has finished running all the tasks.
|
187
205
|
|
188
|
-
|
206
|
+
$ API.run do |on|
|
207
|
+
on.completed { |task_matrix| puts "Done!"}
|
208
|
+
end
|
189
209
|
|
190
210
|
=> Starting workflow...
|
191
211
|
=> starting task A
|
@@ -199,10 +219,10 @@ Using Jongleur is easy:
|
|
199
219
|
=> starting task E
|
200
220
|
=> finished task: E, process: 2506, exit_status: 0, success: true
|
201
221
|
=> Workflow finished
|
222
|
+
=> Done!
|
202
223
|
|
203
224
|
|
204
|
-
|
205
|
-
|
225
|
+
Examples of running Jongleur can be found in the `examples` directory.
|
206
226
|
## Use-Cases
|
207
227
|
### Extract-Transform-Load
|
208
228
|
The ETL workflow is ideally suited to Jongleur. You can define many Extraction tasks -maybe separate Tasks for different data sources- and have them ran in parallel to each other. At the same time Transformation and Loading Tasks wait in turn for the previous task to finish before they start, as in this DAG illustration:
|
@@ -210,7 +230,7 @@ The ETL workflow is ideally suited to Jongleur. You can define many Extraction
|
|
210
230
|
<img src="./bin/img/ETL_DAG.png" width="450" height="450" alt="ETL DAG">
|
211
231
|
|
212
232
|
### Transactions
|
213
|
-
Transactional workflows can be greatly sped up by Jongleur by parallelising parts of the transaction that are usually performed sequentially, i.e
|
233
|
+
Transactional workflows can be greatly sped up by Jongleur by parallelising parts of the transaction that are usually performed sequentially, i.e.:
|
214
234
|
|
215
235
|
<img src="./bin/img/transactional_DAG.png" width="550" height="450" alt="Transaction DAG">
|
216
236
|
|
@@ -224,10 +244,10 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
|
|
224
244
|
## F.A.Q
|
225
245
|
|
226
246
|
### Does Jongleur allow me to pass messages between Tasks?
|
227
|
-
No it doesn't. Each task is run
|
247
|
+
No it doesn't. Each task is run completely independently from the other Tasks. There is no Inter-Process Communication, no common data contexts, no shared memory.
|
228
248
|
|
229
249
|
### How can I share data created by a predecessor Task?
|
230
|
-
This is something that I
|
250
|
+
This is something that I would like to build into Jongleur. For now, you can save a Task's data in a database or KV Store and using the Task's process id as part of the key. Subsequent Tasks can retrieve their predecessor's process ids with
|
231
251
|
|
232
252
|
```
|
233
253
|
API.get_predecessor_pids
|
@@ -239,13 +259,25 @@ and therefore retrieve the data created by those Tasks.
|
|
239
259
|
According to [the official docs](https://ruby-doc.org/core-2.4.3/Process/Status.html) `exit_status` returns the least significant eight bits of the return code of the `stat` call while `success_status` returns true if `stat` is successful.
|
240
260
|
|
241
261
|
### What happens when Jongleur finishes running?
|
242
|
-
When Jongleur finishes running all tasks in its Task Graph -and regardless of whether the Tasks themselves have failed
|
262
|
+
When Jongleur finishes running all tasks in its Task Graph -and regardless of whether the Tasks themselves have failed or not- it will exit the parent process with an exit code of 0.
|
243
263
|
|
244
264
|
### What happens if a Task fails
|
245
|
-
If a Task fails to run or to finish its run, Jongleur will simply go on running any other tasks it can. It will not run any Tasks which depend on the failed Task. The status of the failed Task will be indicated via an appropriate output message and also on the Task Matrix.
|
265
|
+
If a Task fails to run or to finish its run, Jongleur will simply go on running any other tasks it can. It will not run any Tasks which depend on the failed Task. The status of the failed Task will be indicated via an appropriate output message and will also be visible on the Task Matrix.
|
266
|
+
|
267
|
+
### Can I quickly analyse the Task Matrix after Jongleur has finished?
|
268
|
+
Yes. When the `completed` callback is called, Jongleur will enable the following methods:
|
269
|
+
|
270
|
+
```
|
271
|
+
API::successful_tasks
|
272
|
+
API::failed_tasks
|
273
|
+
API::not_ran_tasks
|
274
|
+
API::hung_tasks
|
275
|
+
```
|
276
|
+
|
277
|
+
|
278
|
+
### Are there any execution logs saved?
|
246
279
|
|
247
|
-
|
248
|
-
Jongleur serializes each run's Task Matrix as a JSON file in the `/tmp` directory. You can either view this in an editor or load it and manipulate it in Ruby with
|
280
|
+
Jongleur serializes each run's Task Matrix as a time-stamped JSON file in the `/tmp` directory. You can either view this in an editor or load it and manipulate it in Ruby with
|
249
281
|
|
250
282
|
`JSON.parse( File.read('/tmp/jongleur_task_matrix_08272018_103406.json') )`
|
251
283
|
|
@@ -0,0 +1,49 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
class A < Jongleur::WorkerTask
|
4
|
+
@desc = 'this is task A'
|
5
|
+
def execute
|
6
|
+
sleep 1
|
7
|
+
'A is running... '
|
8
|
+
end
|
9
|
+
end
|
10
|
+
|
11
|
+
class B < Jongleur::WorkerTask
|
12
|
+
@desc = 'this is task B'
|
13
|
+
def execute(colour: 'red', size: 45_634)
|
14
|
+
sleep 3
|
15
|
+
"B is running... with #{colour} and #{size}"
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
class C < Jongleur::WorkerTask
|
20
|
+
@desc = 'this is task C'
|
21
|
+
def execute
|
22
|
+
sleep 1
|
23
|
+
'C is running...'
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
class D < Jongleur::WorkerTask
|
28
|
+
@desc = 'this is task D'
|
29
|
+
def execute
|
30
|
+
sleep 1
|
31
|
+
'D is running...'
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
class E < Jongleur::WorkerTask
|
36
|
+
@desc = 'this is task E'
|
37
|
+
def execute
|
38
|
+
sleep 1
|
39
|
+
'E is running...'
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
class F < Jongleur::WorkerTask
|
44
|
+
@desc = 'this is task F'
|
45
|
+
def execute
|
46
|
+
sleep 1
|
47
|
+
'F is running...'
|
48
|
+
end
|
49
|
+
end
|
data/examples/simple.rb
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
require_relative File.expand_path('../lib/jongleur.rb', __dir__)
|
2
|
+
require_relative './example_tasks'
|
3
|
+
include Jongleur
|
4
|
+
|
5
|
+
test_graph = {
|
6
|
+
A: [:C],
|
7
|
+
B: [:D],
|
8
|
+
C: [:E],
|
9
|
+
D: [:E],
|
10
|
+
E: []
|
11
|
+
}
|
12
|
+
|
13
|
+
API.add_task_graph test_graph
|
14
|
+
|
15
|
+
API.run do |on|
|
16
|
+
puts ">>>> client does something"
|
17
|
+
on.completed do |task_matrix|
|
18
|
+
puts "Jongleur run is complete \n"
|
19
|
+
puts API.successful_tasks(task_matrix)
|
20
|
+
puts "oh-oh" if API.failed_tasks(task_matrix).length > 0
|
21
|
+
# print tasks in order of finishing time
|
22
|
+
task_matrix.sort_by { |x| x.finish_time }.map { |t| STDOUT.puts t.name }
|
23
|
+
end
|
24
|
+
puts ">>>> client does another thing"
|
25
|
+
end
|
data/jongleur.gemspec
CHANGED
@@ -11,7 +11,7 @@ Gem::Specification.new do |spec|
|
|
11
11
|
spec.email = ['fred@bootstrap.me.uk']
|
12
12
|
|
13
13
|
spec.summary = 'A multi-processing task scheduler and manager.'
|
14
|
-
spec.description = 'Launches, schedules and manages tasks represented in a DAG as multiple processes'
|
14
|
+
spec.description = 'Launches, schedules and manages tasks represented in a DAG, as multiple processes'
|
15
15
|
spec.homepage = 'https://gitlab.com/RedFred7/Jongleur'
|
16
16
|
spec.license = 'MIT'
|
17
17
|
|
@@ -24,6 +24,8 @@ Gem::Specification.new do |spec|
|
|
24
24
|
|
25
25
|
spec.add_dependency 'graphviz', '~> 1.1'
|
26
26
|
spec.add_dependency 'os', '~> 1.0'
|
27
|
+
spec.add_dependency 'hollerback', '~> 0.1'
|
28
|
+
|
27
29
|
spec.add_development_dependency 'bundler', '~> 1.16'
|
28
30
|
spec.add_development_dependency 'pry-byebug', '~> 3.4'
|
29
31
|
spec.add_development_dependency 'rake', '~> 10.0'
|
data/lib/jongleur/api.rb
CHANGED
@@ -7,6 +7,7 @@ module Jongleur
|
|
7
7
|
# Here be methods to be accessed by the gem's client, i.e. the public API
|
8
8
|
module API
|
9
9
|
# @!scope class
|
10
|
+
include Hollerback
|
10
11
|
|
11
12
|
# Accepts a task_graph and does some initialisation, namely the assigning
|
12
13
|
# of class variables and creation of the inital task matrix
|
@@ -54,7 +55,7 @@ module Jongleur
|
|
54
55
|
# @return [Array<Jongleur::Task>] a list of Tasks and their current state
|
55
56
|
# @see Jongleur::Task
|
56
57
|
def self.task_matrix
|
57
|
-
@@task_matrix
|
58
|
+
@@task_matrix || []
|
58
59
|
end
|
59
60
|
|
60
61
|
# @!attribute task_graph
|
@@ -123,7 +124,7 @@ module Jongleur
|
|
123
124
|
# all children processes have finished.
|
124
125
|
# @raise [RuntimeError] if there are no implementations for Tasks in the Task Graph
|
125
126
|
# @return [void]
|
126
|
-
def self.run
|
127
|
+
def self.run(&block)
|
127
128
|
unless Implementation.valid_tasks?(task_graph.keys)
|
128
129
|
raise RuntimeError, 'Not all the tasks in the Task Graph are implemented as WorkerTask classes'
|
129
130
|
end
|
@@ -136,13 +137,14 @@ module Jongleur
|
|
136
137
|
begin
|
137
138
|
# with WNOHANG flag we make sure Process.wait is not blocking
|
138
139
|
while (res = Process.wait2(-1, Process::WNOHANG))
|
139
|
-
dead_pid = res
|
140
|
-
|
140
|
+
dead_pid, status = res
|
141
|
+
finish_time = Time.now.to_f
|
141
142
|
dead_task_name = ''
|
142
143
|
Implementation.find_task_by(:pid, dead_pid) do |t|
|
143
144
|
t.running = false
|
144
145
|
t.exit_status = status.exitstatus
|
145
146
|
t.success_status = status.success?
|
147
|
+
t.finish_time = finish_time
|
146
148
|
dead_task_name = t.name
|
147
149
|
end
|
148
150
|
msg = "finished task: %s, process: %i, exit_status: %i, success: %s"
|
@@ -168,17 +170,17 @@ module Jongleur
|
|
168
170
|
end
|
169
171
|
|
170
172
|
loop do
|
171
|
-
# We exit once all the child processes and their descendants are
|
172
|
-
# accounted for
|
173
|
+
# We exit once all the child processes and their descendants are accounted for
|
173
174
|
if Implementation.running_tasks.empty?
|
174
175
|
Implementation.process_message 'Workflow finished'
|
175
176
|
file_name = File.expand_path("jongleur_task_matrix_#{Time.now.strftime('%m%d%Y_%H%M%S')}.json", '/tmp')
|
176
177
|
File.open(file_name, 'w') {|f| f.write(task_matrix.to_json) }
|
178
|
+
hollerback_for(block) { |cb| cb.respond_with(:completed , task_matrix) } if block_given?
|
177
179
|
exit 0
|
178
180
|
end
|
179
181
|
sleep 1
|
180
182
|
end
|
181
|
-
end #method
|
183
|
+
end # method
|
182
184
|
|
183
185
|
|
184
186
|
# Starts all tasks without dependencies as separate processes
|
@@ -133,6 +133,11 @@ module Jongleur
|
|
133
133
|
API.task_matrix.select(&:running)
|
134
134
|
end
|
135
135
|
|
136
|
+
def self.get_task_list
|
137
|
+
API.task_matrix.select(&:running)
|
138
|
+
end
|
139
|
+
|
140
|
+
|
136
141
|
# Find task based on an attribute's value
|
137
142
|
#
|
138
143
|
# @note the methof will find the first matching task. If there are more than one matches,
|
@@ -207,7 +212,8 @@ module Jongleur
|
|
207
212
|
end
|
208
213
|
|
209
214
|
def self.process_message(a_msg)
|
210
|
-
puts(a_msg)
|
215
|
+
STDOUT.puts(a_msg)
|
216
|
+
STDOUT.sync
|
211
217
|
end
|
212
218
|
|
213
219
|
end # module
|
data/lib/jongleur/version.rb
CHANGED
data/lib/jongleur.rb
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
require 'graphviz'
|
3
3
|
require 'json'
|
4
|
+
require 'set'
|
5
|
+
require 'hollerback'
|
4
6
|
require_relative 'jongleur/version'
|
5
7
|
require_relative 'jongleur/helpers'
|
6
8
|
require_relative 'jongleur/worker_task'
|
@@ -23,10 +25,13 @@ module Jongleur
|
|
23
25
|
# @!attribute exit_status
|
24
26
|
# @return [Integer, Nil] the process's return code when the process is exited
|
25
27
|
# Usually 0 for success, 1 for error or Nil otherwise
|
28
|
+
# @!attribute finish_time
|
29
|
+
# @return [Float, 0] the timestamp of process completion as a floating point number
|
30
|
+
# of seconds since the Epoch
|
26
31
|
# @!attribute success_status
|
27
32
|
# @return [Boolean, Nil] true if process finished successfully, false if it didn't
|
28
33
|
# or nil if process didn't exit properly.
|
29
|
-
Task = Struct.new(:name, :pid, :running, :exit_status, :success_status)
|
34
|
+
Task = Struct.new(:name, :pid, :running, :exit_status, :finish_time, :success_status)
|
30
35
|
|
31
36
|
$stdout.sync = true
|
32
37
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: jongleur
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0
|
4
|
+
version: 1.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Fred Heath
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-
|
11
|
+
date: 2018-12-31 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: graphviz
|
@@ -38,6 +38,20 @@ dependencies:
|
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
40
|
version: '1.0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: hollerback
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0.1'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0.1'
|
41
55
|
- !ruby/object:Gem::Dependency
|
42
56
|
name: bundler
|
43
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -136,12 +150,11 @@ dependencies:
|
|
136
150
|
- - "~>"
|
137
151
|
- !ruby/object:Gem::Version
|
138
152
|
version: '0.9'
|
139
|
-
description: Launches, schedules and manages tasks represented in a DAG as multiple
|
153
|
+
description: Launches, schedules and manages tasks represented in a DAG, as multiple
|
140
154
|
processes
|
141
155
|
email:
|
142
156
|
- fred@bootstrap.me.uk
|
143
|
-
executables:
|
144
|
-
- test_gem.sh
|
157
|
+
executables: []
|
145
158
|
extensions: []
|
146
159
|
extra_rdoc_files: []
|
147
160
|
files:
|
@@ -160,7 +173,8 @@ files:
|
|
160
173
|
- bin/img/jongleur_m-2015.jpg
|
161
174
|
- bin/img/transactional_DAG.png
|
162
175
|
- bin/setup
|
163
|
-
-
|
176
|
+
- examples/example_tasks.rb
|
177
|
+
- examples/simple.rb
|
164
178
|
- jongleur.gemspec
|
165
179
|
- lib/jongleur.rb
|
166
180
|
- lib/jongleur/api.rb
|
data/exe/test_gem.sh
DELETED