taskinator 0.4.0 → 0.4.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/ISSUE_TEMPLATE/bug_report.md +38 -0
- data/.github/ISSUE_TEMPLATE/custom.md +10 -0
- data/.github/ISSUE_TEMPLATE/feature_request.md +20 -0
- data/.github/workflows/taskinator.yml +41 -0
- data/.ruby-version +1 -1
- data/CHANGELOG.md +19 -0
- data/CODE_OF_CONDUCT.md +128 -0
- data/CONTRIBUTING.md +58 -0
- data/Gemfile +4 -0
- data/Gemfile.lock +71 -22
- data/README.md +184 -71
- data/lib/taskinator/api.rb +14 -5
- data/lib/taskinator/definition/builder.rb +16 -7
- data/lib/taskinator/persistence.rb +3 -0
- data/lib/taskinator/queues/active_job.rb +53 -0
- data/lib/taskinator/queues.rb +1 -0
- data/lib/taskinator/version.rb +1 -1
- data/lib/taskinator.rb +2 -2
- data/spec/spec_helper.rb +14 -0
- data/spec/support/test_flows.rb +38 -0
- data/spec/taskinator/api_spec.rb +16 -0
- data/spec/taskinator/definition/builder_spec.rb +48 -0
- data/spec/taskinator/persistence_spec.rb +21 -17
- data/spec/taskinator/queues/active_job_spec.rb +80 -0
- data/spec/taskinator/task_spec.rb +1 -1
- metadata +12 -5
- data/.coveralls.yml +0 -1
- data/.travis.yml +0 -23
data/README.md
CHANGED
@@ -1,25 +1,29 @@
|
|
1
1
|
# Taskinator
|
2
2
|
|
3
3
|
[![Gem Version](https://badge.fury.io/rb/taskinator.svg)](http://badge.fury.io/rb/taskinator)
|
4
|
-
[![Build Status](https://
|
4
|
+
[![Build Status](https://img.shields.io/github/workflow/status/virtualstaticvoid/taskinator/Taskinator?style=flat-square)](https://github.com/virtualstaticvoid/taskinator/actions)
|
5
5
|
[![Code Climate](https://codeclimate.com/github/virtualstaticvoid/taskinator.png)](https://codeclimate.com/github/virtualstaticvoid/taskinator)
|
6
|
-
[![Coverage Status](https://coveralls.io/repos/virtualstaticvoid/taskinator/badge.png)](https://coveralls.io/r/virtualstaticvoid/taskinator)
|
7
6
|
|
8
|
-
A simple orchestration library for running complex processes or workflows in Ruby.
|
9
|
-
|
10
|
-
for
|
7
|
+
A simple orchestration library for running complex processes or workflows in Ruby.
|
8
|
+
Processes are defined using a simple DSL, where the sequences and tasks are defined.
|
9
|
+
Processes can then be queued for execution. Sequences can be synchronous or asynchronous,
|
10
|
+
and the overall process can be monitored for completion or failure.
|
11
11
|
|
12
|
-
Processes and tasks are executed by background workers and you can use any one of the
|
12
|
+
Processes and tasks are executed by background workers and you can use any one of the
|
13
|
+
following gems:
|
13
14
|
|
15
|
+
* [active_job](https://github.com/rails/rails/tree/main/activejob)
|
14
16
|
* [resque](https://github.com/resque/resque)
|
15
17
|
* [sidekiq](https://github.com/mperham/sidekiq)
|
16
18
|
* [delayed_job](https://github.com/collectiveidea/delayed_job)
|
17
19
|
|
18
|
-
The configuration and state of each process and their respective tasks is stored using
|
20
|
+
The configuration and state of each process and their respective tasks is stored using
|
21
|
+
Redis key/values.
|
19
22
|
|
20
23
|
## Requirements
|
21
24
|
|
22
|
-
The latest MRI
|
25
|
+
The latest MRI 2.x or 3.x version. Other versions/VMs are untested, but might work fine.
|
26
|
+
MRI 1.x is not supported.
|
23
27
|
|
24
28
|
Redis 2.4 or greater is required.
|
25
29
|
|
@@ -35,7 +39,7 @@ Add this line to your application's Gemfile:
|
|
35
39
|
|
36
40
|
And then execute:
|
37
41
|
|
38
|
-
$ bundle
|
42
|
+
$ bundle install
|
39
43
|
|
40
44
|
Or install it yourself as:
|
41
45
|
|
@@ -49,6 +53,7 @@ Start by creating a "process" module and extending `Taskinator::Definition`.
|
|
49
53
|
|
50
54
|
```ruby
|
51
55
|
require 'taskinator'
|
56
|
+
|
52
57
|
module MyProcess
|
53
58
|
extend Taskinator::Definition
|
54
59
|
|
@@ -68,7 +73,8 @@ module MyProcess
|
|
68
73
|
end
|
69
74
|
```
|
70
75
|
|
71
|
-
The `define_process` method optionally takes the list of expected arguments which are used
|
76
|
+
The `define_process` method optionally takes the list of expected arguments which are used
|
77
|
+
to validate the arguments supplied when creating a new process.
|
72
78
|
These should be specified with symbols.
|
73
79
|
|
74
80
|
```ruby
|
@@ -87,7 +93,8 @@ process = MyProcess.create_process Date.today, :option_1 => true
|
|
87
93
|
|
88
94
|
_NOTE:_ The current implementation performs a naive check on the count of arguments.
|
89
95
|
|
90
|
-
Next, specify the tasks with their corresponding implementation methods, that make up the
|
96
|
+
Next, specify the tasks with their corresponding implementation methods, that make up the
|
97
|
+
process, using the `task` method and providing the `method` to execute for the task.
|
91
98
|
|
92
99
|
```ruby
|
93
100
|
module MyProcess
|
@@ -108,7 +115,8 @@ module MyProcess
|
|
108
115
|
end
|
109
116
|
```
|
110
117
|
|
111
|
-
More complex processes may define sequential or concurrent steps, using the `sequential`
|
118
|
+
More complex processes may define sequential or concurrent steps, using the `sequential`
|
119
|
+
and `concurrent` methods respectively.
|
112
120
|
|
113
121
|
```ruby
|
114
122
|
module MyProcess
|
@@ -141,10 +149,16 @@ module MyProcess
|
|
141
149
|
end
|
142
150
|
```
|
143
151
|
|
144
|
-
|
152
|
+
#### Reusing ActiveJob jobs
|
153
|
+
|
154
|
+
It is likely that you already have one or more [jobs](https://guides.rubyonrails.org/active_job_basics.html)
|
155
|
+
and want to reuse them within the process definition.
|
156
|
+
|
157
|
+
Define a `job` step, providing the class of the Active Job to run and then taskinator will
|
158
|
+
invoke that job as part of the process.
|
145
159
|
|
146
|
-
|
147
|
-
|
160
|
+
The `job` step will be queued and executed on same queue as
|
161
|
+
[configured by the job](https://guides.rubyonrails.org/active_job_basics.html#queues).
|
148
162
|
|
149
163
|
```ruby
|
150
164
|
# E.g. A resque worker
|
@@ -168,8 +182,13 @@ module MyProcess
|
|
168
182
|
end
|
169
183
|
```
|
170
184
|
|
171
|
-
|
172
|
-
|
185
|
+
#### Data Driven Process Definitions
|
186
|
+
|
187
|
+
You can also define data driven tasks using the `for_each` method, which takes an iterator method
|
188
|
+
name as an argument.
|
189
|
+
|
190
|
+
The iterator method yields the parameters necessary for the task or job. Notice that the task
|
191
|
+
method takes a parameter in this case, which will be the return values provided by the iterator.
|
173
192
|
|
174
193
|
```ruby
|
175
194
|
module MyProcess
|
@@ -192,8 +211,11 @@ module MyProcess
|
|
192
211
|
end
|
193
212
|
```
|
194
213
|
|
195
|
-
|
196
|
-
|
214
|
+
#### Branching
|
215
|
+
|
216
|
+
It is possible to branch the process logic based on the options hash passed in when creating
|
217
|
+
a process. The `options?` method takes the options key as an argument and calls the supplied
|
218
|
+
block if the option is present and it's value is _truthy_.
|
197
219
|
|
198
220
|
```ruby
|
199
221
|
module MyProcess
|
@@ -227,8 +249,13 @@ process2 = MyProcess.create_process
|
|
227
249
|
process2.tasks.count #=> 1
|
228
250
|
```
|
229
251
|
|
230
|
-
|
231
|
-
|
252
|
+
#### Transformations
|
253
|
+
|
254
|
+
In addition, it is possible to transform the arguments used by a task or job, by including
|
255
|
+
a `transform` step in the definition.
|
256
|
+
|
257
|
+
Similarly for the `for_each` method, `transform` takes a method name as an argument.
|
258
|
+
The transformer method must yield the new arguments as required.
|
232
259
|
|
233
260
|
```ruby
|
234
261
|
module MyProcess
|
@@ -252,6 +279,8 @@ module MyProcess
|
|
252
279
|
end
|
253
280
|
```
|
254
281
|
|
282
|
+
#### Subprocesses
|
283
|
+
|
255
284
|
Processes can be composed of other processes too:
|
256
285
|
|
257
286
|
```ruby
|
@@ -273,7 +302,10 @@ module MyProcess
|
|
273
302
|
end
|
274
303
|
```
|
275
304
|
|
276
|
-
|
305
|
+
#### Complex Process Definitions
|
306
|
+
|
307
|
+
Any combination or nesting of `task`, `sequential`, `concurrent` and `for_each` steps are
|
308
|
+
possible. E.g.
|
277
309
|
|
278
310
|
```ruby
|
279
311
|
module MyProcess
|
@@ -306,13 +338,17 @@ module MyProcess
|
|
306
338
|
end
|
307
339
|
```
|
308
340
|
|
309
|
-
In this example, the `work_step_begin` is executed, followed by the `work_step_all_at_once`
|
310
|
-
the sub process `MySubProcess` is created and
|
341
|
+
In this example, the `work_step_begin` is executed, followed by the `work_step_all_at_once`
|
342
|
+
steps which are executed concurrently, then the sub process `MySubProcess` is created and
|
343
|
+
executed, followed by the `work_step_one_by_one` tasks which are executed sequentially and
|
311
344
|
finally the `work_step_end` is executed.
|
312
345
|
|
313
|
-
It is also possible to embed conditional logic within the process definition stages in
|
314
|
-
|
315
|
-
|
346
|
+
It is also possible to embed conditional logic within the process definition stages in
|
347
|
+
order to produce steps based on the required logic.
|
348
|
+
|
349
|
+
All builder methods are available within the scope of the `define_process` block. These
|
350
|
+
methods include `args` and `options` which are passed into the `create_process` method
|
351
|
+
of the definition.
|
316
352
|
|
317
353
|
E.g.
|
318
354
|
|
@@ -332,7 +368,8 @@ module MyProcess
|
|
332
368
|
end
|
333
369
|
|
334
370
|
# when creating this proces, you supply to option when calling `create_process`
|
335
|
-
# in this example, 'args' will be an array [1,2,3]
|
371
|
+
# in this example, 'args' will be an array [1,2,3]
|
372
|
+
# and options will be a Hash {:send_notification => true}
|
336
373
|
MyProcess.create_process(1, 2, 3, :send_notification => true)
|
337
374
|
|
338
375
|
```
|
@@ -364,8 +401,12 @@ To best understand how arguments are handled, you need to break it down into 3 p
|
|
364
401
|
* Creation and
|
365
402
|
* Execution
|
366
403
|
|
367
|
-
Firstly, a process definition is declarative in that the `define_process` and a mix of
|
368
|
-
|
404
|
+
Firstly, a process definition is declarative in that the `define_process` and a mix of
|
405
|
+
`sequential`, `concurrent`, `for_each`, `task` and `job` directives provide the way to
|
406
|
+
specify the sequencing of the steps for the process.
|
407
|
+
|
408
|
+
Taskinator will interprete this definition and execute each step in the desired sequence
|
409
|
+
or concurrency.
|
369
410
|
|
370
411
|
Consider the following process definition:
|
371
412
|
|
@@ -411,11 +452,17 @@ end
|
|
411
452
|
|
412
453
|
There are three tasks; namely `:work_step_1`, `:work_step_2` and `:work_step_3`.
|
413
454
|
|
414
|
-
The third task, `:work_step_3`, is built up using the `for_each` iterator, which means that
|
455
|
+
The third task, `:work_step_3`, is built up using the `for_each` iterator, which means that
|
456
|
+
the number of `:work_step_3` tasks will depend on how many times the `additional_step`
|
457
|
+
iterator method yields to the definition.
|
458
|
+
|
459
|
+
This brings us to the creation part. When `create_process` is called on the given module,
|
460
|
+
you provide arguments to it, which will get passed onto the respective `task` and
|
461
|
+
`for_each` iterator methods.
|
415
462
|
|
416
|
-
|
463
|
+
So, considering the `MySimpleProcess` module shown above, `work_step_1`, `work_step_2`
|
464
|
+
and `work_step_3` methods each expect arguments.
|
417
465
|
|
418
|
-
So, considering the `MySimpleProcess` module shown above, `work_step_1`, `work_step_2` and `work_step_3` methods each expect arguments.
|
419
466
|
These will ultimately come from the arguments passed into the `create_process` method.
|
420
467
|
|
421
468
|
E.g.
|
@@ -438,7 +485,8 @@ process = MySimpleProcess.create_process(options)
|
|
438
485
|
|
439
486
|
```
|
440
487
|
|
441
|
-
To best understand how the process is created, consider the following "procedural" code
|
488
|
+
To best understand how the process is created, consider the following "procedural" code
|
489
|
+
for how it could work.
|
442
490
|
|
443
491
|
```ruby
|
444
492
|
# A process, which maps the target and a list of steps
|
@@ -535,11 +583,17 @@ process.execute
|
|
535
583
|
|
536
584
|
```
|
537
585
|
|
538
|
-
In reality, each task is executed by a worker process, possibly on another host, so the
|
586
|
+
In reality, each task is executed by a worker process, possibly on another host, so the
|
587
|
+
execution process isn't as simple, but this example should help you to understand
|
588
|
+
conceptually how the process is executed, and how the arguments are propagated through.
|
539
589
|
|
540
590
|
### Monitoring
|
541
591
|
|
542
|
-
|
592
|
+
NOTE: This aspect of the library is still a work in progress.
|
593
|
+
|
594
|
+
#### Processes
|
595
|
+
|
596
|
+
To monitor the state of the processes, use the `Taskinator::Api::Processes` class.
|
543
597
|
|
544
598
|
```ruby
|
545
599
|
processes = Taskinator::Api::Processes.new
|
@@ -549,11 +603,57 @@ processes.each do |process|
|
|
549
603
|
end
|
550
604
|
```
|
551
605
|
|
606
|
+
#### Debugging
|
607
|
+
|
608
|
+
To aid debugging specific processes and tasks, where the process or task identifier is
|
609
|
+
known, it is possible to retrieve the specific task or process using `Taskinator::Api`.
|
610
|
+
|
611
|
+
To retrieve a specific process, given the process identifier:
|
612
|
+
|
613
|
+
```ruby
|
614
|
+
process_id = "SUPPLY-PROCESS-IDENTIFIER"
|
615
|
+
process = Taskinator::Api.find_process(process_id)
|
616
|
+
|
617
|
+
puts process.inspect
|
618
|
+
puts process.current_state
|
619
|
+
puts process.tasks
|
620
|
+
# etc...
|
621
|
+
```
|
622
|
+
|
623
|
+
The type of process may be one of the following:
|
624
|
+
|
625
|
+
* `Taskinator::Process::Sequential`
|
626
|
+
* `Taskinator::Process::Concurrent`
|
627
|
+
|
628
|
+
Then, to retrieve a specific task, given the task identifier:
|
629
|
+
|
630
|
+
```ruby
|
631
|
+
task_id = "SUPPLY-TASK-IDENTIFIER"
|
632
|
+
task = Taskinator::Api.find_task(task_id)
|
633
|
+
|
634
|
+
puts task.inspect
|
635
|
+
puts task.class
|
636
|
+
puts task.args # for Step and Job types
|
637
|
+
puts task.sub_process.tasks # for SubProcess type
|
638
|
+
# etc...
|
639
|
+
```
|
640
|
+
|
641
|
+
Depending on the type of task, different attributes will be available for inspection.
|
642
|
+
|
643
|
+
The types include:
|
644
|
+
|
645
|
+
* `Taskinator::Task::Step`
|
646
|
+
* `Taskinator::Task::Job`
|
647
|
+
* `Taskinator::Task::SubProcess`
|
648
|
+
|
552
649
|
## Configuration
|
553
650
|
|
554
651
|
### Redis
|
555
652
|
|
556
|
-
By default Taskinator assumes Redis is located at `localhost:6397`. This is fine for development,
|
653
|
+
By default Taskinator assumes Redis is located at `localhost:6397`. This is fine for development,
|
654
|
+
but for many production environments you will need to point to an external Redis server.
|
655
|
+
You may also what to use a namespace for the Redis keys.
|
656
|
+
|
557
657
|
_NOTE:_ The configuration hash _must_ have symbolized keys.
|
558
658
|
|
559
659
|
```ruby
|
@@ -568,15 +668,19 @@ end
|
|
568
668
|
Or, alternatively, via an `ENV` variable
|
569
669
|
|
570
670
|
Set the `REDIS_PROVIDER` environment variable to the Redis server url.
|
571
|
-
E.g. On Heroku, with RedisGreen: set REDIS_PROVIDER=REDISGREEN_URL and Taskinator will use the
|
671
|
+
E.g. On Heroku, with RedisGreen: set REDIS_PROVIDER=REDISGREEN_URL and Taskinator will use the
|
672
|
+
value of the `REDISGREEN_URL` environment variable when connecting to Redis.
|
572
673
|
|
573
674
|
You may also use the generic `REDIS_URL` which may be set to your own private Redis server.
|
574
675
|
|
575
|
-
The Redis configuration leverages the same setup as `sidekiq`. For advanced options, checkout the
|
676
|
+
The Redis configuration leverages the same setup as `sidekiq`. For advanced options, checkout the
|
677
|
+
[Sidekiq Advanced Options](https://github.com/mperham/sidekiq/wiki/Advanced-Options#complete-control)
|
678
|
+
wiki page for more information.
|
576
679
|
|
577
680
|
### Queues
|
578
681
|
|
579
|
-
By default the queue names for process and task workers is `default`, however, you can specify
|
682
|
+
By default the queue names for process and task workers is `default`, however, you can specify
|
683
|
+
the queue names as follows:
|
580
684
|
|
581
685
|
```ruby
|
582
686
|
Taskinator.configure do |config|
|
@@ -589,7 +693,8 @@ end
|
|
589
693
|
|
590
694
|
### Instrumentation
|
591
695
|
|
592
|
-
It is possible to instrument processes, tasks and jobs by providing an instrumeter such
|
696
|
+
It is possible to instrument processes, tasks and jobs by providing an instrumeter such
|
697
|
+
as `ActiveSupport::Notifications`.
|
593
698
|
|
594
699
|
```ruby
|
595
700
|
Taskinator.configure do |config|
|
@@ -607,40 +712,41 @@ end
|
|
607
712
|
|
608
713
|
The following instrumentation events are issued:
|
609
714
|
|
610
|
-
| Event
|
611
|
-
|
612
|
-
| `taskinator.process.created`
|
613
|
-
| `taskinator.process.saved`
|
614
|
-
| `taskinator.process.enqueued`
|
615
|
-
| `taskinator.process.processing`
|
616
|
-
| `taskinator.process.paused`
|
617
|
-
| `taskinator.process.resumed`
|
618
|
-
| `taskinator.process.completed`
|
619
|
-
| `taskinator.process.cancelled`
|
620
|
-
| `taskinator.process.failed`
|
621
|
-
| `taskinator.task.enqueued`
|
622
|
-
| `taskinator.task.processing`
|
623
|
-
| `taskinator.task.completed`
|
624
|
-
| `taskinator.task.cancelled`
|
625
|
-
| `taskinator.task.failed`
|
715
|
+
| Event | When |
|
716
|
+
|---------------------------------|----------------------------------------------------------|
|
717
|
+
| `taskinator.process.created` | After a root process gets created |
|
718
|
+
| `taskinator.process.saved` | After a root process has been persisted to Redis |
|
719
|
+
| `taskinator.process.enqueued` | After a process or subprocess is enqueued for processing |
|
720
|
+
| `taskinator.process.processing` | When a process or subprocess is processing |
|
721
|
+
| `taskinator.process.paused` | When a process or subprocess is paused |
|
722
|
+
| `taskinator.process.resumed` | When a process or subprocess is resumed |
|
723
|
+
| `taskinator.process.completed` | After a process or subprocess has completed processing |
|
724
|
+
| `taskinator.process.cancelled` | After a process or subprocess has been cancelled |
|
725
|
+
| `taskinator.process.failed` | After a process or subprocess has failed |
|
726
|
+
| `taskinator.task.enqueued` | After a task has been enqueued |
|
727
|
+
| `taskinator.task.processing` | When a task is processing |
|
728
|
+
| `taskinator.task.completed` | After a task has completed |
|
729
|
+
| `taskinator.task.cancelled` | After a task has been cancelled |
|
730
|
+
| `taskinator.task.failed` | After a task has failed |
|
626
731
|
|
627
732
|
For all events, the data included contains the following information:
|
628
733
|
|
629
|
-
| Key
|
630
|
-
|
631
|
-
| `:type`
|
632
|
-
| `:process_uuid`
|
633
|
-
| `:process_options`
|
634
|
-
| `:uuid`
|
635
|
-
| `:options`
|
636
|
-
| `:state`
|
637
|
-
| `:percentage_completed`
|
638
|
-
| `:percentage_failed`
|
639
|
-
| `:percentage_cancelled`
|
734
|
+
| Key | Value |
|
735
|
+
|---------------------------------|----------------------------------------------------------|
|
736
|
+
| `:type` | The type name of the component reporting the event |
|
737
|
+
| `:process_uuid` | The UUID of the root process |
|
738
|
+
| `:process_options` | Options hash of the root process |
|
739
|
+
| `:uuid` | The UUID of the respective task, job or sub process |
|
740
|
+
| `:options` | Options hash of the component |
|
741
|
+
| `:state` | State of the component |
|
742
|
+
| `:percentage_completed` | The percentage of completed tasks |
|
743
|
+
| `:percentage_failed` | The percentage of failed tasks |
|
744
|
+
| `:percentage_cancelled` | The percentage of cancelled tasks |
|
640
745
|
|
641
746
|
## Notes
|
642
747
|
|
643
|
-
The persistence logic is decoupled from the implementation, so it is possible to implement
|
748
|
+
The persistence logic is decoupled from the implementation, so it is possible to implement
|
749
|
+
another backing store if required.
|
644
750
|
|
645
751
|
## Contributing
|
646
752
|
|
@@ -651,12 +757,19 @@ The persistence logic is decoupled from the implementation, so it is possible to
|
|
651
757
|
5. Create new Pull Request
|
652
758
|
|
653
759
|
## License
|
760
|
+
|
654
761
|
MIT Copyright (c) 2014 Chris Stefano
|
655
762
|
|
656
763
|
Portions of code are from the Sidekiq project, Copyright (c) Contributed Systems LLC.
|
657
764
|
|
658
765
|
## Inspiration
|
659
766
|
|
660
|
-
Inspired by the [sidekiq](https://github.com/mperham/sidekiq) and
|
767
|
+
Inspired by the [sidekiq](https://github.com/mperham/sidekiq) and
|
768
|
+
[workflow](https://github.com/geekq/workflow) gems.
|
769
|
+
|
770
|
+
For other workflow solutions, checkout [Stonepath](https://github.com/bokmann/stonepath),
|
771
|
+
the now deprecated [ruote](https://github.com/jmettraux/ruote) gem and
|
772
|
+
[workflow](https://github.com/geekq/workflow).
|
661
773
|
|
662
|
-
|
774
|
+
Alternatively, for a robust enterprise ready solution checkout the
|
775
|
+
[AWS Flow Framework for Ruby](http://docs.aws.amazon.com/amazonswf/latest/awsrbflowguide/welcome.html).
|
data/lib/taskinator/api.rb
CHANGED
@@ -13,12 +13,13 @@ module Taskinator
|
|
13
13
|
def each(&block)
|
14
14
|
return to_enum(__method__) unless block_given?
|
15
15
|
|
16
|
+
identifiers = Taskinator.redis do |conn|
|
17
|
+
conn.smembers(@processes_list_key)
|
18
|
+
end
|
19
|
+
|
16
20
|
instance_cache = {}
|
17
|
-
|
18
|
-
|
19
|
-
uuids.each do |uuid|
|
20
|
-
yield Process.fetch(uuid, instance_cache)
|
21
|
-
end
|
21
|
+
identifiers.each do |identifier|
|
22
|
+
yield Process.fetch(identifier, instance_cache)
|
22
23
|
end
|
23
24
|
end
|
24
25
|
|
@@ -28,5 +29,13 @@ module Taskinator
|
|
28
29
|
end
|
29
30
|
end
|
30
31
|
end
|
32
|
+
|
33
|
+
def self.find_process(identifier)
|
34
|
+
Process.fetch(identifier)
|
35
|
+
end
|
36
|
+
|
37
|
+
def self.find_task(identifier)
|
38
|
+
Task.fetch(identifier)
|
39
|
+
end
|
31
40
|
end
|
32
41
|
end
|
@@ -24,7 +24,10 @@ module Taskinator
|
|
24
24
|
raise ArgumentError, 'block' unless block_given?
|
25
25
|
|
26
26
|
sub_process = Process.define_sequential_process_for(@definition, options)
|
27
|
-
|
27
|
+
task = define_sub_process_task(@process, sub_process, options)
|
28
|
+
Builder.new(sub_process, @definition, *@args).instance_eval(&block)
|
29
|
+
@process.tasks << task if sub_process.tasks.any?
|
30
|
+
nil
|
28
31
|
end
|
29
32
|
|
30
33
|
# defines a sub process of tasks which are executed concurrently
|
@@ -32,7 +35,10 @@ module Taskinator
|
|
32
35
|
raise ArgumentError, 'block' unless block_given?
|
33
36
|
|
34
37
|
sub_process = Process.define_concurrent_process_for(@definition, complete_on, options)
|
35
|
-
|
38
|
+
task = define_sub_process_task(@process, sub_process, options)
|
39
|
+
Builder.new(sub_process, @definition, *@args).instance_eval(&block)
|
40
|
+
@process.tasks << task if sub_process.tasks.any?
|
41
|
+
nil
|
36
42
|
end
|
37
43
|
|
38
44
|
# dynamically defines tasks, using the given @iterator method
|
@@ -51,6 +57,7 @@ module Taskinator
|
|
51
57
|
@executor.send(method, *method_args) do |*args|
|
52
58
|
Builder.new(@process, @definition, *args).instance_eval(&block)
|
53
59
|
end
|
60
|
+
nil
|
54
61
|
end
|
55
62
|
|
56
63
|
alias_method :transform, :for_each
|
@@ -61,6 +68,7 @@ module Taskinator
|
|
61
68
|
raise NoMethodError, method unless @executor.respond_to?(method)
|
62
69
|
|
63
70
|
define_step_task(@process, method, @args, options)
|
71
|
+
nil
|
64
72
|
end
|
65
73
|
|
66
74
|
# defines a task which executes the given @job
|
@@ -70,6 +78,7 @@ module Taskinator
|
|
70
78
|
raise ArgumentError, 'job' unless job.methods.include?(:perform) || job.instance_methods.include?(:perform)
|
71
79
|
|
72
80
|
define_job_task(@process, job, @args, options)
|
81
|
+
nil
|
73
82
|
end
|
74
83
|
|
75
84
|
# defines a sub process task, for the given @definition
|
@@ -82,7 +91,10 @@ module Taskinator
|
|
82
91
|
# TODO: decide whether the sub process to dynamically receive arguments
|
83
92
|
|
84
93
|
sub_process = definition.create_sub_process(*@args, combine_options(options))
|
85
|
-
|
94
|
+
task = define_sub_process_task(@process, sub_process, options)
|
95
|
+
Builder.new(sub_process, definition, *@args)
|
96
|
+
@process.tasks << task if sub_process.tasks.any?
|
97
|
+
nil
|
86
98
|
end
|
87
99
|
|
88
100
|
private
|
@@ -100,10 +112,7 @@ module Taskinator
|
|
100
112
|
end
|
101
113
|
|
102
114
|
def define_sub_process_task(process, sub_process, options={})
|
103
|
-
|
104
|
-
Task.define_sub_process_task(process, sub_process, combine_options(options))
|
105
|
-
}
|
106
|
-
sub_process
|
115
|
+
Task.define_sub_process_task(process, sub_process, combine_options(options))
|
107
116
|
end
|
108
117
|
|
109
118
|
def define_task(process)
|
@@ -604,6 +604,9 @@ module Taskinator
|
|
604
604
|
end
|
605
605
|
|
606
606
|
def visit_tasks(tasks)
|
607
|
+
@conn.expire "#{@key}:tasks", expire_in
|
608
|
+
@conn.expire "#{@key}.count", expire_in
|
609
|
+
@conn.expire "#{@key}.pending", expire_in
|
607
610
|
tasks.each do |task|
|
608
611
|
RedisCleanupVisitor.new(@conn, task, expire_in).visit
|
609
612
|
end
|
@@ -0,0 +1,53 @@
|
|
1
|
+
module Taskinator
|
2
|
+
module Queues
|
3
|
+
|
4
|
+
# https://guides.rubyonrails.org/active_job_basics.html
|
5
|
+
|
6
|
+
def self.create_active_job_adapter(config={})
|
7
|
+
ActiveJobAdapter.new(config)
|
8
|
+
end
|
9
|
+
|
10
|
+
class ActiveJobAdapter
|
11
|
+
def initialize(config={})
|
12
|
+
@config = Taskinator::Queues::DefaultConfig.merge(config)
|
13
|
+
end
|
14
|
+
|
15
|
+
def enqueue_create_process(definition, uuid, args)
|
16
|
+
queue = definition.queue || @config[:definition_queue]
|
17
|
+
CreateProcessWorker.set(:queue => queue)
|
18
|
+
.perform_later(definition.name, uuid, Taskinator::Persistence.serialize(args))
|
19
|
+
end
|
20
|
+
|
21
|
+
def enqueue_process(process)
|
22
|
+
queue = process.queue || @config[:process_queue]
|
23
|
+
ProcessWorker.set(:queue => queue)
|
24
|
+
.perform_later(process.uuid)
|
25
|
+
end
|
26
|
+
|
27
|
+
def enqueue_task(task)
|
28
|
+
queue = task.queue || @config[:task_queue]
|
29
|
+
TaskWorker.set(:queue => queue)
|
30
|
+
.perform_later(task.uuid)
|
31
|
+
end
|
32
|
+
|
33
|
+
class CreateProcessWorker < ApplicationJob
|
34
|
+
def perform(definition_name, uuid, args)
|
35
|
+
Taskinator::CreateProcessWorker.new(definition_name, uuid, args).perform
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
39
|
+
class ProcessWorker < ApplicationJob
|
40
|
+
def perform(process_uuid)
|
41
|
+
Taskinator::ProcessWorker.new(process_uuid).perform
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
class TaskWorker < ApplicationJob
|
46
|
+
def perform(task_uuid)
|
47
|
+
Taskinator::TaskWorker.new(task_uuid).perform
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
data/lib/taskinator/queues.rb
CHANGED
@@ -48,6 +48,7 @@ module Taskinator
|
|
48
48
|
end
|
49
49
|
end
|
50
50
|
|
51
|
+
require 'taskinator/queues/active_job' if defined?(ApplicationJob)
|
51
52
|
require 'taskinator/queues/delayed_job' if defined?(Delayed)
|
52
53
|
require 'taskinator/queues/resque' if defined?(Resque)
|
53
54
|
require 'taskinator/queues/sidekiq' if defined?(Sidekiq)
|
data/lib/taskinator/version.rb
CHANGED
data/lib/taskinator.rb
CHANGED
@@ -98,7 +98,7 @@ module Taskinator
|
|
98
98
|
|
99
99
|
# the queue adapter to use
|
100
100
|
# supported adapters include
|
101
|
-
# :delayed_job, :redis and :sidekiq
|
101
|
+
# :active_job, :delayed_job, :redis and :sidekiq
|
102
102
|
# NOTE: ensure that the respective gem is included
|
103
103
|
attr_reader :queue_adapter
|
104
104
|
|
@@ -118,7 +118,7 @@ module Taskinator
|
|
118
118
|
|
119
119
|
def queue
|
120
120
|
@queue ||= begin
|
121
|
-
adapter = self.queue_adapter || :resque
|
121
|
+
adapter = self.queue_adapter || :resque # TODO: change default to :active_job
|
122
122
|
config = queue_config || {}
|
123
123
|
Taskinator::Queues.create_adapter(adapter, config)
|
124
124
|
end
|