taskinator 0.4.1 → 0.4.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.md +38 -0
- data/.github/ISSUE_TEMPLATE/custom.md +10 -0
- data/.github/ISSUE_TEMPLATE/feature_request.md +20 -0
- data/.github/workflows/taskinator.yml +41 -0
- data/.ruby-version +1 -1
- data/CHANGELOG.md +14 -0
- data/CODE_OF_CONDUCT.md +128 -0
- data/CONTRIBUTING.md +58 -0
- data/Gemfile +4 -0
- data/Gemfile.lock +75 -26
- data/README.md +241 -75
- data/lib/taskinator/api.rb +14 -5
- data/lib/taskinator/persistence.rb +3 -0
- data/lib/taskinator/queues/active_job.rb +53 -0
- data/lib/taskinator/queues.rb +1 -0
- data/lib/taskinator/version.rb +1 -1
- data/lib/taskinator.rb +2 -2
- data/spec/spec_helper.rb +14 -0
- data/spec/support/test_flows.rb +38 -0
- data/spec/taskinator/api_spec.rb +16 -0
- data/spec/taskinator/persistence_spec.rb +21 -17
- data/spec/taskinator/queues/active_job_spec.rb +80 -0
- metadata +12 -5
- data/.coveralls.yml +0 -1
- data/.travis.yml +0 -23
data/README.md
CHANGED
@@ -1,25 +1,29 @@
|
|
1
1
|
# Taskinator
|
2
2
|
|
3
3
|
[![Gem Version](https://badge.fury.io/rb/taskinator.svg)](http://badge.fury.io/rb/taskinator)
|
4
|
-
[![Build Status](https://
|
4
|
+
[![Build Status](https://img.shields.io/github/workflow/status/virtualstaticvoid/taskinator/Taskinator?style=flat-square)](https://github.com/virtualstaticvoid/taskinator/actions)
|
5
5
|
[![Code Climate](https://codeclimate.com/github/virtualstaticvoid/taskinator.png)](https://codeclimate.com/github/virtualstaticvoid/taskinator)
|
6
|
-
[![Coverage Status](https://coveralls.io/repos/virtualstaticvoid/taskinator/badge.png)](https://coveralls.io/r/virtualstaticvoid/taskinator)
|
7
6
|
|
8
|
-
A simple orchestration library for running complex processes or workflows in Ruby.
|
9
|
-
|
10
|
-
for
|
7
|
+
A simple orchestration library for running complex processes or workflows in Ruby.
|
8
|
+
Processes are defined using a simple DSL, where the sequences and tasks are defined.
|
9
|
+
Processes can then be queued for execution. Sequences can be synchronous or asynchronous,
|
10
|
+
and the overall process can be monitored for completion or failure.
|
11
11
|
|
12
|
-
Processes and tasks are executed by background workers and you can use any one of the
|
12
|
+
Processes and tasks are executed by background workers and you can use any one of the
|
13
|
+
following gems:
|
13
14
|
|
15
|
+
* [active_job](https://github.com/rails/rails/tree/main/activejob)
|
14
16
|
* [resque](https://github.com/resque/resque)
|
15
17
|
* [sidekiq](https://github.com/mperham/sidekiq)
|
16
18
|
* [delayed_job](https://github.com/collectiveidea/delayed_job)
|
17
19
|
|
18
|
-
The configuration and state of each process and their respective tasks is stored using
|
20
|
+
The configuration and state of each process and their respective tasks is stored using
|
21
|
+
Redis key/values.
|
19
22
|
|
20
23
|
## Requirements
|
21
24
|
|
22
|
-
The latest MRI
|
25
|
+
The latest MRI 2.x or 3.x version. Other versions/VMs are untested, but might work fine.
|
26
|
+
MRI 1.x is not supported.
|
23
27
|
|
24
28
|
Redis 2.4 or greater is required.
|
25
29
|
|
@@ -35,12 +39,34 @@ Add this line to your application's Gemfile:
|
|
35
39
|
|
36
40
|
And then execute:
|
37
41
|
|
38
|
-
$ bundle
|
42
|
+
$ bundle install
|
39
43
|
|
40
44
|
Or install it yourself as:
|
41
45
|
|
42
46
|
$ gem install taskinator
|
43
47
|
|
48
|
+
If you are using Taskinator within a Rails application, then add an initializer, such as
|
49
|
+
`config/initializers/taskinator.rb`, with the following configuration content:
|
50
|
+
|
51
|
+
```ruby
|
52
|
+
# config/initializers/taskinator.rb
|
53
|
+
Taskinator.configure do |config|
|
54
|
+
|
55
|
+
# configure the queue adapter to use
|
56
|
+
# can be :active_job, :delayed_job, :redis or :sidekiq
|
57
|
+
config.queue_adapter = :redis
|
58
|
+
|
59
|
+
# configure redis
|
60
|
+
config.redis = {
|
61
|
+
:url => 'redis://redis.example.com:7372/12',
|
62
|
+
:namespace => 'mynamespace'
|
63
|
+
}
|
64
|
+
|
65
|
+
end
|
66
|
+
```
|
67
|
+
|
68
|
+
See the configuration section below for more configuration details.
|
69
|
+
|
44
70
|
## Usage
|
45
71
|
|
46
72
|
### Definition
|
@@ -49,6 +75,7 @@ Start by creating a "process" module and extending `Taskinator::Definition`.
|
|
49
75
|
|
50
76
|
```ruby
|
51
77
|
require 'taskinator'
|
78
|
+
|
52
79
|
module MyProcess
|
53
80
|
extend Taskinator::Definition
|
54
81
|
|
@@ -68,7 +95,8 @@ module MyProcess
|
|
68
95
|
end
|
69
96
|
```
|
70
97
|
|
71
|
-
The `define_process` method optionally takes the list of expected arguments which are used
|
98
|
+
The `define_process` method optionally takes the list of expected arguments which are used
|
99
|
+
to validate the arguments supplied when creating a new process.
|
72
100
|
These should be specified with symbols.
|
73
101
|
|
74
102
|
```ruby
|
@@ -87,7 +115,8 @@ process = MyProcess.create_process Date.today, :option_1 => true
|
|
87
115
|
|
88
116
|
_NOTE:_ The current implementation performs a naive check on the count of arguments.
|
89
117
|
|
90
|
-
Next, specify the tasks with their corresponding implementation methods, that make up the
|
118
|
+
Next, specify the tasks with their corresponding implementation methods, that make up the
|
119
|
+
process, using the `task` method and providing the `method` to execute for the task.
|
91
120
|
|
92
121
|
```ruby
|
93
122
|
module MyProcess
|
@@ -108,7 +137,8 @@ module MyProcess
|
|
108
137
|
end
|
109
138
|
```
|
110
139
|
|
111
|
-
More complex processes may define sequential or concurrent steps, using the `sequential`
|
140
|
+
More complex processes may define sequential or concurrent steps, using the `sequential`
|
141
|
+
and `concurrent` methods respectively.
|
112
142
|
|
113
143
|
```ruby
|
114
144
|
module MyProcess
|
@@ -141,10 +171,16 @@ module MyProcess
|
|
141
171
|
end
|
142
172
|
```
|
143
173
|
|
144
|
-
|
174
|
+
#### Reusing ActiveJob jobs
|
145
175
|
|
146
|
-
|
147
|
-
|
176
|
+
It is likely that you already have one or more [jobs](https://guides.rubyonrails.org/active_job_basics.html)
|
177
|
+
and want to reuse them within the process definition.
|
178
|
+
|
179
|
+
Define a `job` step, providing the class of the Active Job to run and then taskinator will
|
180
|
+
invoke that job as part of the process.
|
181
|
+
|
182
|
+
The `job` step will be queued and executed on same queue as
|
183
|
+
[configured by the job](https://guides.rubyonrails.org/active_job_basics.html#queues).
|
148
184
|
|
149
185
|
```ruby
|
150
186
|
# E.g. A resque worker
|
@@ -168,8 +204,13 @@ module MyProcess
|
|
168
204
|
end
|
169
205
|
```
|
170
206
|
|
171
|
-
|
172
|
-
|
207
|
+
#### Data Driven Process Definitions
|
208
|
+
|
209
|
+
You can also define data driven tasks using the `for_each` method, which takes an iterator method
|
210
|
+
name as an argument.
|
211
|
+
|
212
|
+
The iterator method yields the parameters necessary for the task or job. Notice that the task
|
213
|
+
method takes a parameter in this case, which will be the return values provided by the iterator.
|
173
214
|
|
174
215
|
```ruby
|
175
216
|
module MyProcess
|
@@ -192,8 +233,11 @@ module MyProcess
|
|
192
233
|
end
|
193
234
|
```
|
194
235
|
|
195
|
-
|
196
|
-
|
236
|
+
#### Branching
|
237
|
+
|
238
|
+
It is possible to branch the process logic based on the options hash passed in when creating
|
239
|
+
a process. The `options?` method takes the options key as an argument and calls the supplied
|
240
|
+
block if the option is present and it's value is _truthy_.
|
197
241
|
|
198
242
|
```ruby
|
199
243
|
module MyProcess
|
@@ -227,8 +271,13 @@ process2 = MyProcess.create_process
|
|
227
271
|
process2.tasks.count #=> 1
|
228
272
|
```
|
229
273
|
|
230
|
-
|
231
|
-
|
274
|
+
#### Transformations
|
275
|
+
|
276
|
+
In addition, it is possible to transform the arguments used by a task or job, by including
|
277
|
+
a `transform` step in the definition.
|
278
|
+
|
279
|
+
Similarly for the `for_each` method, `transform` takes a method name as an argument.
|
280
|
+
The transformer method must yield the new arguments as required.
|
232
281
|
|
233
282
|
```ruby
|
234
283
|
module MyProcess
|
@@ -252,6 +301,8 @@ module MyProcess
|
|
252
301
|
end
|
253
302
|
```
|
254
303
|
|
304
|
+
#### Subprocesses
|
305
|
+
|
255
306
|
Processes can be composed of other processes too:
|
256
307
|
|
257
308
|
```ruby
|
@@ -273,7 +324,10 @@ module MyProcess
|
|
273
324
|
end
|
274
325
|
```
|
275
326
|
|
276
|
-
|
327
|
+
#### Complex Process Definitions
|
328
|
+
|
329
|
+
Any combination or nesting of `task`, `sequential`, `concurrent` and `for_each` steps are
|
330
|
+
possible. E.g.
|
277
331
|
|
278
332
|
```ruby
|
279
333
|
module MyProcess
|
@@ -306,13 +360,17 @@ module MyProcess
|
|
306
360
|
end
|
307
361
|
```
|
308
362
|
|
309
|
-
In this example, the `work_step_begin` is executed, followed by the `work_step_all_at_once`
|
310
|
-
the sub process `MySubProcess` is created and
|
363
|
+
In this example, the `work_step_begin` is executed, followed by the `work_step_all_at_once`
|
364
|
+
steps which are executed concurrently, then the sub process `MySubProcess` is created and
|
365
|
+
executed, followed by the `work_step_one_by_one` tasks which are executed sequentially and
|
311
366
|
finally the `work_step_end` is executed.
|
312
367
|
|
313
|
-
It is also possible to embed conditional logic within the process definition stages in
|
314
|
-
|
315
|
-
|
368
|
+
It is also possible to embed conditional logic within the process definition stages in
|
369
|
+
order to produce steps based on the required logic.
|
370
|
+
|
371
|
+
All builder methods are available within the scope of the `define_process` block. These
|
372
|
+
methods include `args` and `options` which are passed into the `create_process` method
|
373
|
+
of the definition.
|
316
374
|
|
317
375
|
E.g.
|
318
376
|
|
@@ -332,7 +390,8 @@ module MyProcess
|
|
332
390
|
end
|
333
391
|
|
334
392
|
# when creating this proces, you supply to option when calling `create_process`
|
335
|
-
# in this example, 'args' will be an array [1,2,3]
|
393
|
+
# in this example, 'args' will be an array [1,2,3]
|
394
|
+
# and options will be a Hash {:send_notification => true}
|
336
395
|
MyProcess.create_process(1, 2, 3, :send_notification => true)
|
337
396
|
|
338
397
|
```
|
@@ -364,8 +423,12 @@ To best understand how arguments are handled, you need to break it down into 3 p
|
|
364
423
|
* Creation and
|
365
424
|
* Execution
|
366
425
|
|
367
|
-
Firstly, a process definition is declarative in that the `define_process` and a mix of
|
368
|
-
|
426
|
+
Firstly, a process definition is declarative in that the `define_process` and a mix of
|
427
|
+
`sequential`, `concurrent`, `for_each`, `task` and `job` directives provide the way to
|
428
|
+
specify the sequencing of the steps for the process.
|
429
|
+
|
430
|
+
Taskinator will interprete this definition and execute each step in the desired sequence
|
431
|
+
or concurrency.
|
369
432
|
|
370
433
|
Consider the following process definition:
|
371
434
|
|
@@ -411,11 +474,17 @@ end
|
|
411
474
|
|
412
475
|
There are three tasks; namely `:work_step_1`, `:work_step_2` and `:work_step_3`.
|
413
476
|
|
414
|
-
The third task, `:work_step_3`, is built up using the `for_each` iterator, which means that
|
477
|
+
The third task, `:work_step_3`, is built up using the `for_each` iterator, which means that
|
478
|
+
the number of `:work_step_3` tasks will depend on how many times the `additional_step`
|
479
|
+
iterator method yields to the definition.
|
480
|
+
|
481
|
+
This brings us to the creation part. When `create_process` is called on the given module,
|
482
|
+
you provide arguments to it, which will get passed onto the respective `task` and
|
483
|
+
`for_each` iterator methods.
|
415
484
|
|
416
|
-
|
485
|
+
So, considering the `MySimpleProcess` module shown above, `work_step_1`, `work_step_2`
|
486
|
+
and `work_step_3` methods each expect arguments.
|
417
487
|
|
418
|
-
So, considering the `MySimpleProcess` module shown above, `work_step_1`, `work_step_2` and `work_step_3` methods each expect arguments.
|
419
488
|
These will ultimately come from the arguments passed into the `create_process` method.
|
420
489
|
|
421
490
|
E.g.
|
@@ -438,7 +507,8 @@ process = MySimpleProcess.create_process(options)
|
|
438
507
|
|
439
508
|
```
|
440
509
|
|
441
|
-
To best understand how the process is created, consider the following "procedural" code
|
510
|
+
To best understand how the process is created, consider the following "procedural" code
|
511
|
+
for how it could work.
|
442
512
|
|
443
513
|
```ruby
|
444
514
|
# A process, which maps the target and a list of steps
|
@@ -535,11 +605,17 @@ process.execute
|
|
535
605
|
|
536
606
|
```
|
537
607
|
|
538
|
-
In reality, each task is executed by a worker process, possibly on another host, so the
|
608
|
+
In reality, each task is executed by a worker process, possibly on another host, so the
|
609
|
+
execution process isn't as simple, but this example should help you to understand
|
610
|
+
conceptually how the process is executed, and how the arguments are propagated through.
|
539
611
|
|
540
612
|
### Monitoring
|
541
613
|
|
542
|
-
|
614
|
+
NOTE: This aspect of the library is still a work in progress.
|
615
|
+
|
616
|
+
#### Processes
|
617
|
+
|
618
|
+
To monitor the state of the processes, use the `Taskinator::Api::Processes` class.
|
543
619
|
|
544
620
|
```ruby
|
545
621
|
processes = Taskinator::Api::Processes.new
|
@@ -549,51 +625,130 @@ processes.each do |process|
|
|
549
625
|
end
|
550
626
|
```
|
551
627
|
|
628
|
+
#### Debugging
|
629
|
+
|
630
|
+
To aid debugging specific processes and tasks, where the process or task identifier is
|
631
|
+
known, it is possible to retrieve the specific task or process using `Taskinator::Api`.
|
632
|
+
|
633
|
+
To retrieve a specific process, given the process identifier:
|
634
|
+
|
635
|
+
```ruby
|
636
|
+
process_id = "SUPPLY-PROCESS-IDENTIFIER"
|
637
|
+
process = Taskinator::Api.find_process(process_id)
|
638
|
+
|
639
|
+
puts process.inspect
|
640
|
+
puts process.current_state
|
641
|
+
puts process.tasks
|
642
|
+
# etc...
|
643
|
+
```
|
644
|
+
|
645
|
+
The type of process may be one of the following:
|
646
|
+
|
647
|
+
* `Taskinator::Process::Sequential`
|
648
|
+
* `Taskinator::Process::Concurrent`
|
649
|
+
|
650
|
+
Then, to retrieve a specific task, given the task identifier:
|
651
|
+
|
652
|
+
```ruby
|
653
|
+
task_id = "SUPPLY-TASK-IDENTIFIER"
|
654
|
+
task = Taskinator::Api.find_task(task_id)
|
655
|
+
|
656
|
+
puts task.inspect
|
657
|
+
puts task.class
|
658
|
+
puts task.args # for Step and Job types
|
659
|
+
puts task.sub_process.tasks # for SubProcess type
|
660
|
+
# etc...
|
661
|
+
```
|
662
|
+
|
663
|
+
Depending on the type of task, different attributes will be available for inspection.
|
664
|
+
|
665
|
+
The types include:
|
666
|
+
|
667
|
+
* `Taskinator::Task::Step`
|
668
|
+
* `Taskinator::Task::Job`
|
669
|
+
* `Taskinator::Task::SubProcess`
|
670
|
+
|
552
671
|
## Configuration
|
553
672
|
|
554
673
|
### Redis
|
555
674
|
|
556
|
-
By default Taskinator assumes Redis is located at `localhost:6397`. This is fine for development,
|
675
|
+
By default Taskinator assumes Redis is located at `localhost:6397`. This is fine for development,
|
676
|
+
but for many production environments you will need to point to an external Redis server.
|
677
|
+
You may also what to use a namespace for the Redis keys.
|
678
|
+
|
557
679
|
_NOTE:_ The configuration hash _must_ have symbolized keys.
|
558
680
|
|
559
681
|
```ruby
|
560
682
|
Taskinator.configure do |config|
|
561
|
-
|
562
|
-
|
563
|
-
|
564
|
-
|
683
|
+
|
684
|
+
# redis configuration
|
685
|
+
config.redis = {
|
686
|
+
:url => 'redis://redis.example.com:7372/12',
|
687
|
+
:namespace => 'mynamespace'
|
688
|
+
}
|
689
|
+
|
565
690
|
end
|
566
691
|
```
|
567
692
|
|
568
693
|
Or, alternatively, via an `ENV` variable
|
569
694
|
|
570
695
|
Set the `REDIS_PROVIDER` environment variable to the Redis server url.
|
571
|
-
E.g. On Heroku, with RedisGreen: set REDIS_PROVIDER=REDISGREEN_URL and Taskinator will use the
|
696
|
+
E.g. On Heroku, with RedisGreen: set `REDIS_PROVIDER=REDISGREEN_URL` and Taskinator will use the
|
697
|
+
value of the `REDISGREEN_URL` environment variable when connecting to Redis.
|
572
698
|
|
573
699
|
You may also use the generic `REDIS_URL` which may be set to your own private Redis server.
|
574
700
|
|
575
|
-
The Redis configuration leverages the same setup as `sidekiq`. For advanced options, checkout the
|
701
|
+
The Redis configuration leverages the same setup as `sidekiq`. For advanced options, checkout the
|
702
|
+
[Sidekiq Advanced Options](https://github.com/mperham/sidekiq/wiki/Advanced-Options#complete-control)
|
703
|
+
wiki page for more information.
|
576
704
|
|
577
705
|
### Queues
|
578
706
|
|
579
|
-
|
707
|
+
To configure the queue adapter to use, set `config.queue_adapter` to one of the following values:
|
708
|
+
|
709
|
+
* `:active_job`
|
710
|
+
* `:delayed_job`
|
711
|
+
* `:redis`
|
712
|
+
* `:sidekiq`
|
713
|
+
|
714
|
+
As follows:
|
580
715
|
|
581
716
|
```ruby
|
582
717
|
Taskinator.configure do |config|
|
718
|
+
|
719
|
+
# configure the queue adapter to use
|
720
|
+
# can be :active_job, :delayed_job, :redis or :sidekiq
|
721
|
+
config.queue_adapter = :redis
|
722
|
+
|
723
|
+
end
|
724
|
+
```
|
725
|
+
|
726
|
+
By default the queue names for process and task workers is `default`, however, you can specify
|
727
|
+
the queue names as follows:
|
728
|
+
|
729
|
+
```ruby
|
730
|
+
Taskinator.configure do |config|
|
731
|
+
|
732
|
+
# queue configuration
|
583
733
|
config.queue_config = {
|
584
734
|
:process_queue => :default,
|
585
735
|
:task_queue => :default
|
586
736
|
}
|
737
|
+
|
587
738
|
end
|
588
739
|
```
|
589
740
|
|
590
741
|
### Instrumentation
|
591
742
|
|
592
|
-
It is possible to instrument processes, tasks and jobs by providing an instrumeter such
|
743
|
+
It is possible to instrument processes, tasks and jobs by providing an instrumeter such
|
744
|
+
as `ActiveSupport::Notifications`.
|
593
745
|
|
594
746
|
```ruby
|
595
747
|
Taskinator.configure do |config|
|
748
|
+
|
749
|
+
# configure instrumenter to use
|
596
750
|
config.instrumenter = ActiveSupport::Notifications
|
751
|
+
|
597
752
|
end
|
598
753
|
```
|
599
754
|
|
@@ -601,46 +756,50 @@ Alternatively, you can use the built-in instrumenter for logging to the console
|
|
601
756
|
|
602
757
|
```ruby
|
603
758
|
Taskinator.configure do |config|
|
759
|
+
|
760
|
+
# configure instrumenter to use
|
604
761
|
config.instrumenter = Taskinator::ConsoleInstrumenter.new
|
762
|
+
|
605
763
|
end
|
606
764
|
```
|
607
765
|
|
608
766
|
The following instrumentation events are issued:
|
609
767
|
|
610
|
-
| Event
|
611
|
-
|
612
|
-
| `taskinator.process.created`
|
613
|
-
| `taskinator.process.saved`
|
614
|
-
| `taskinator.process.enqueued`
|
615
|
-
| `taskinator.process.processing`
|
616
|
-
| `taskinator.process.paused`
|
617
|
-
| `taskinator.process.resumed`
|
618
|
-
| `taskinator.process.completed`
|
619
|
-
| `taskinator.process.cancelled`
|
620
|
-
| `taskinator.process.failed`
|
621
|
-
| `taskinator.task.enqueued`
|
622
|
-
| `taskinator.task.processing`
|
623
|
-
| `taskinator.task.completed`
|
624
|
-
| `taskinator.task.cancelled`
|
625
|
-
| `taskinator.task.failed`
|
768
|
+
| Event | When |
|
769
|
+
|---------------------------------|----------------------------------------------------------|
|
770
|
+
| `taskinator.process.created` | After a root process gets created |
|
771
|
+
| `taskinator.process.saved` | After a root process has been persisted to Redis |
|
772
|
+
| `taskinator.process.enqueued` | After a process or subprocess is enqueued for processing |
|
773
|
+
| `taskinator.process.processing` | When a process or subprocess is processing |
|
774
|
+
| `taskinator.process.paused` | When a process or subprocess is paused |
|
775
|
+
| `taskinator.process.resumed` | When a process or subprocess is resumed |
|
776
|
+
| `taskinator.process.completed` | After a process or subprocess has completed processing |
|
777
|
+
| `taskinator.process.cancelled` | After a process or subprocess has been cancelled |
|
778
|
+
| `taskinator.process.failed` | After a process or subprocess has failed |
|
779
|
+
| `taskinator.task.enqueued` | After a task has been enqueued |
|
780
|
+
| `taskinator.task.processing` | When a task is processing |
|
781
|
+
| `taskinator.task.completed` | After a task has completed |
|
782
|
+
| `taskinator.task.cancelled` | After a task has been cancelled |
|
783
|
+
| `taskinator.task.failed` | After a task has failed |
|
626
784
|
|
627
785
|
For all events, the data included contains the following information:
|
628
786
|
|
629
|
-
| Key
|
630
|
-
|
631
|
-
| `:type`
|
632
|
-
| `:process_uuid`
|
633
|
-
| `:process_options`
|
634
|
-
| `:uuid`
|
635
|
-
| `:options`
|
636
|
-
| `:state`
|
637
|
-
| `:percentage_completed`
|
638
|
-
| `:percentage_failed`
|
639
|
-
| `:percentage_cancelled`
|
787
|
+
| Key | Value |
|
788
|
+
|---------------------------------|----------------------------------------------------------|
|
789
|
+
| `:type` | The type name of the component reporting the event |
|
790
|
+
| `:process_uuid` | The UUID of the root process |
|
791
|
+
| `:process_options` | Options hash of the root process |
|
792
|
+
| `:uuid` | The UUID of the respective task, job or sub process |
|
793
|
+
| `:options` | Options hash of the component |
|
794
|
+
| `:state` | State of the component |
|
795
|
+
| `:percentage_completed` | The percentage of completed tasks |
|
796
|
+
| `:percentage_failed` | The percentage of failed tasks |
|
797
|
+
| `:percentage_cancelled` | The percentage of cancelled tasks |
|
640
798
|
|
641
799
|
## Notes
|
642
800
|
|
643
|
-
The persistence logic is decoupled from the implementation, so it is possible to implement
|
801
|
+
The persistence logic is decoupled from the implementation, so it is possible to implement
|
802
|
+
another backing store if required.
|
644
803
|
|
645
804
|
## Contributing
|
646
805
|
|
@@ -651,12 +810,19 @@ The persistence logic is decoupled from the implementation, so it is possible to
|
|
651
810
|
5. Create new Pull Request
|
652
811
|
|
653
812
|
## License
|
813
|
+
|
654
814
|
MIT Copyright (c) 2014 Chris Stefano
|
655
815
|
|
656
816
|
Portions of code are from the Sidekiq project, Copyright (c) Contributed Systems LLC.
|
657
817
|
|
658
818
|
## Inspiration
|
659
819
|
|
660
|
-
Inspired by the [sidekiq](https://github.com/mperham/sidekiq) and
|
820
|
+
Inspired by the [sidekiq](https://github.com/mperham/sidekiq) and
|
821
|
+
[workflow](https://github.com/geekq/workflow) gems.
|
822
|
+
|
823
|
+
For other workflow solutions, checkout [Stonepath](https://github.com/bokmann/stonepath),
|
824
|
+
the now deprecated [ruote](https://github.com/jmettraux/ruote) gem and
|
825
|
+
[workflow](https://github.com/geekq/workflow).
|
661
826
|
|
662
|
-
|
827
|
+
Alternatively, for a robust enterprise ready solution checkout the
|
828
|
+
[AWS Flow Framework for Ruby](http://docs.aws.amazon.com/amazonswf/latest/awsrbflowguide/welcome.html).
|
data/lib/taskinator/api.rb
CHANGED
@@ -13,12 +13,13 @@ module Taskinator
|
|
13
13
|
def each(&block)
|
14
14
|
return to_enum(__method__) unless block_given?
|
15
15
|
|
16
|
+
identifiers = Taskinator.redis do |conn|
|
17
|
+
conn.smembers(@processes_list_key)
|
18
|
+
end
|
19
|
+
|
16
20
|
instance_cache = {}
|
17
|
-
|
18
|
-
|
19
|
-
uuids.each do |uuid|
|
20
|
-
yield Process.fetch(uuid, instance_cache)
|
21
|
-
end
|
21
|
+
identifiers.each do |identifier|
|
22
|
+
yield Process.fetch(identifier, instance_cache)
|
22
23
|
end
|
23
24
|
end
|
24
25
|
|
@@ -28,5 +29,13 @@ module Taskinator
|
|
28
29
|
end
|
29
30
|
end
|
30
31
|
end
|
32
|
+
|
33
|
+
def self.find_process(identifier)
|
34
|
+
Process.fetch(identifier)
|
35
|
+
end
|
36
|
+
|
37
|
+
def self.find_task(identifier)
|
38
|
+
Task.fetch(identifier)
|
39
|
+
end
|
31
40
|
end
|
32
41
|
end
|
@@ -604,6 +604,9 @@ module Taskinator
|
|
604
604
|
end
|
605
605
|
|
606
606
|
def visit_tasks(tasks)
|
607
|
+
@conn.expire "#{@key}:tasks", expire_in
|
608
|
+
@conn.expire "#{@key}.count", expire_in
|
609
|
+
@conn.expire "#{@key}.pending", expire_in
|
607
610
|
tasks.each do |task|
|
608
611
|
RedisCleanupVisitor.new(@conn, task, expire_in).visit
|
609
612
|
end
|
@@ -0,0 +1,53 @@
|
|
1
|
+
module Taskinator
|
2
|
+
module Queues
|
3
|
+
|
4
|
+
# https://guides.rubyonrails.org/active_job_basics.html
|
5
|
+
|
6
|
+
def self.create_active_job_adapter(config={})
|
7
|
+
ActiveJobAdapter.new(config)
|
8
|
+
end
|
9
|
+
|
10
|
+
class ActiveJobAdapter
|
11
|
+
def initialize(config={})
|
12
|
+
@config = Taskinator::Queues::DefaultConfig.merge(config)
|
13
|
+
end
|
14
|
+
|
15
|
+
def enqueue_create_process(definition, uuid, args)
|
16
|
+
queue = definition.queue || @config[:definition_queue]
|
17
|
+
CreateProcessWorker.set(:queue => queue)
|
18
|
+
.perform_later(definition.name, uuid, Taskinator::Persistence.serialize(args))
|
19
|
+
end
|
20
|
+
|
21
|
+
def enqueue_process(process)
|
22
|
+
queue = process.queue || @config[:process_queue]
|
23
|
+
ProcessWorker.set(:queue => queue)
|
24
|
+
.perform_later(process.uuid)
|
25
|
+
end
|
26
|
+
|
27
|
+
def enqueue_task(task)
|
28
|
+
queue = task.queue || @config[:task_queue]
|
29
|
+
TaskWorker.set(:queue => queue)
|
30
|
+
.perform_later(task.uuid)
|
31
|
+
end
|
32
|
+
|
33
|
+
class CreateProcessWorker < ApplicationJob
|
34
|
+
def perform(definition_name, uuid, args)
|
35
|
+
Taskinator::CreateProcessWorker.new(definition_name, uuid, args).perform
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
39
|
+
class ProcessWorker < ApplicationJob
|
40
|
+
def perform(process_uuid)
|
41
|
+
Taskinator::ProcessWorker.new(process_uuid).perform
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
class TaskWorker < ApplicationJob
|
46
|
+
def perform(task_uuid)
|
47
|
+
Taskinator::TaskWorker.new(task_uuid).perform
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
data/lib/taskinator/queues.rb
CHANGED
@@ -48,6 +48,7 @@ module Taskinator
|
|
48
48
|
end
|
49
49
|
end
|
50
50
|
|
51
|
+
require 'taskinator/queues/active_job' if defined?(ApplicationJob)
|
51
52
|
require 'taskinator/queues/delayed_job' if defined?(Delayed)
|
52
53
|
require 'taskinator/queues/resque' if defined?(Resque)
|
53
54
|
require 'taskinator/queues/sidekiq' if defined?(Sidekiq)
|
data/lib/taskinator/version.rb
CHANGED
data/lib/taskinator.rb
CHANGED
@@ -98,7 +98,7 @@ module Taskinator
|
|
98
98
|
|
99
99
|
# the queue adapter to use
|
100
100
|
# supported adapters include
|
101
|
-
# :delayed_job, :redis and :sidekiq
|
101
|
+
# :active_job, :delayed_job, :redis and :sidekiq
|
102
102
|
# NOTE: ensure that the respective gem is included
|
103
103
|
attr_reader :queue_adapter
|
104
104
|
|
@@ -118,7 +118,7 @@ module Taskinator
|
|
118
118
|
|
119
119
|
def queue
|
120
120
|
@queue ||= begin
|
121
|
-
adapter = self.queue_adapter || :resque
|
121
|
+
adapter = self.queue_adapter || :resque # TODO: change default to :active_job
|
122
122
|
config = queue_config || {}
|
123
123
|
Taskinator::Queues.create_adapter(adapter, config)
|
124
124
|
end
|