sidekiq-workflow 0.0.1 → 0.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/CODE_OF_CONDUCT.md +0 -0
- data/Gemfile +0 -0
- data/LICENSE.txt +0 -0
- data/README.md +70 -2
- data/Rakefile +0 -0
- data/dump.rdb +0 -0
- data/lib/sidekiq/workflow.rb +43 -5
- data/lib/sidekiq/workflow/client_middleware.rb +32 -0
- data/lib/sidekiq/workflow/manager.rb +41 -0
- data/lib/sidekiq/workflow/monitor.rb +11 -0
- data/lib/sidekiq/workflow/node.rb +14 -0
- data/lib/sidekiq/workflow/railtie.rb +0 -0
- data/lib/sidekiq/workflow/server_middleware.rb +23 -0
- data/lib/sidekiq/workflow/version.rb +1 -1
- data/sidekiq-workflow.gemspec +5 -1
- metadata +55 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c896e5dd3c9e174a591c2485eef0c2f8f993847f
|
4
|
+
data.tar.gz: d8d07782138963a6773a9ac6491316173a067aaa
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ac3613923dca6a2881b4a75a3eae57adcaf118e88584af49fd1b24dacb7c7d991f2afc0d504396131cac1a03b76e4ebd32f8ca073d7ea7fead948171fa8470ee
|
7
|
+
data.tar.gz: 3a4f57e72957c252ff73f9f369caed3d9e53c11957769bd4c51b1bdfef498cd2c5fbe3115d975621c0e5d3d9e0896fbb6b21959cbaf503d347d1f3b3c8adb617
|
data/.gitignore
CHANGED
data/CODE_OF_CONDUCT.md
CHANGED
File without changes
|
data/Gemfile
CHANGED
File without changes
|
data/LICENSE.txt
CHANGED
File without changes
|
data/README.md
CHANGED
@@ -1,4 +1,7 @@
|
|
1
1
|
# Sidekiq::Workflow
|
2
|
+
_sidekiq-workflow is currently in development and is not yet suited to
|
3
|
+
production use. Please continue at your own risk...YOU HAVE BEEN
|
4
|
+
WARNED!_
|
2
5
|
|
3
6
|
## Installation
|
4
7
|
|
@@ -18,7 +21,72 @@ Or install it yourself as:
|
|
18
21
|
|
19
22
|
## Usage
|
20
23
|
|
21
|
-
|
24
|
+
To run a workflow, sidekiq-workflow provides a simple interface:
|
25
|
+
|
26
|
+
```
|
27
|
+
Sidekiq::Workflow.run(sequence_definition)
|
28
|
+
```
|
29
|
+
|
30
|
+
A sequence definition is composed of an array-defined workflow. The
|
31
|
+
convention here is that jobs within the root array element are executed
|
32
|
+
sequentially, one level deeper is executed in parallel. Currently,
|
33
|
+
sidekiq-workflow only supports simple embedded parallelism at one level.
|
34
|
+
This means that it only supports workflows like the following:
|
35
|
+
|
36
|
+
```
|
37
|
+
worker_2
|
38
|
+
/ \
|
39
|
+
worker_1 worker_4
|
40
|
+
\ /
|
41
|
+
worker_3
|
42
|
+
```
|
43
|
+
A workflow like this is easily defined with the following structure:
|
44
|
+
```ruby
|
45
|
+
[
|
46
|
+
worker_1,
|
47
|
+
[worker_2, worker_3],
|
48
|
+
worker_4
|
49
|
+
]
|
50
|
+
```
|
51
|
+
workers in the root array are processed sequentially and any workers in nested
|
52
|
+
arrays are processes in parrallel. The `worker_#` values above can be a
|
53
|
+
strings or worker class names so:
|
54
|
+
|
55
|
+
```ruby
|
56
|
+
[
|
57
|
+
"AccountWorker",
|
58
|
+
["UserWorker", "ProfileWorker"],
|
59
|
+
"EmailWorker"
|
60
|
+
]
|
61
|
+
```
|
62
|
+
|
63
|
+
is equivalent to
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
[
|
67
|
+
AccountWorker,
|
68
|
+
[UserWorker, ProfileWorker],
|
69
|
+
EmailWorker
|
70
|
+
]
|
71
|
+
```
|
72
|
+
|
73
|
+
## Coming Soon
|
74
|
+
|
75
|
+
Parameter passing and custom worker methods will be coming shortly as
|
76
|
+
well as expanded workflow structure providing for any mixture of
|
77
|
+
sequential and parallel definitions to make workflows like this
|
78
|
+
possible:
|
79
|
+
|
80
|
+
```
|
81
|
+
worker_5
|
82
|
+
/ \
|
83
|
+
worker_4 ----worker_7
|
84
|
+
/ \ / /
|
85
|
+
worker_1 worker_6 /
|
86
|
+
\ /
|
87
|
+
worker_2---worker_3
|
88
|
+
|
89
|
+
```
|
22
90
|
|
23
91
|
## Development
|
24
92
|
|
@@ -30,7 +98,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
|
|
30
98
|
|
31
99
|
## Contributing
|
32
100
|
|
33
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
101
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/michaelkelly322/sidekiq-workflow. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
34
102
|
|
35
103
|
Please also take a look at [this post](https://badmonkeydev.wordpress.com/2015/05/23/code-review-best-practices/) for guidlines on submitting pull requests. Thanks!
|
36
104
|
|
data/Rakefile
CHANGED
File without changes
|
data/dump.rdb
ADDED
Binary file
|
data/lib/sidekiq/workflow.rb
CHANGED
@@ -1,11 +1,49 @@
|
|
1
1
|
require "sidekiq"
|
2
|
+
require "redis-namespace"
|
3
|
+
require "dm-core"
|
4
|
+
require "dm-types"
|
5
|
+
require "dm-redis-adapter"
|
2
6
|
require "sidekiq/workflow/version"
|
7
|
+
require "sidekiq/workflow/client_middleware"
|
8
|
+
require "sidekiq/workflow/server_middleware"
|
9
|
+
require "sidekiq/workflow/manager"
|
10
|
+
require "sidekiq/workflow/node"
|
3
11
|
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
12
|
+
class TestWorker
|
13
|
+
include Sidekiq::Worker
|
14
|
+
|
15
|
+
def perform
|
16
|
+
sleep(5)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
DataMapper.setup(:default, {:adapter => "redis"})
|
21
|
+
|
22
|
+
module Sidekiq::Workflow
|
23
|
+
|
24
|
+
def self.run(workflow_definition)
|
25
|
+
workflow = Workflow.create(workflow_definition)
|
26
|
+
|
27
|
+
Manager.execute workflow
|
28
|
+
workflow.id
|
29
|
+
end
|
30
|
+
|
31
|
+
class Workflow
|
32
|
+
include DataMapper::Resource
|
33
|
+
|
34
|
+
property :id, Serial
|
35
|
+
property :workflow_definition, Json
|
36
|
+
property :nodes, Json
|
37
|
+
|
38
|
+
def initialize(workflow_definition)
|
39
|
+
raise ArgumentError.new('workflow_definition can\'t be nil') unless workflow_definition
|
40
|
+
self.workflow_definition = workflow_definition
|
41
|
+
self.nodes = parsed_definition
|
42
|
+
binding.pry
|
43
|
+
end
|
44
|
+
|
45
|
+
def parsed_definition
|
46
|
+
@parsed_definition ||= Manager.parse workflow_definition
|
9
47
|
end
|
10
48
|
end
|
11
49
|
end
|
@@ -0,0 +1,32 @@
|
|
1
|
+
class Sidekiq::Workflow::ClientMiddleware
|
2
|
+
attr_accessor :wid, :nid
|
3
|
+
|
4
|
+
def call(worker_class, msg, queue, redis_pool)
|
5
|
+
puts "BEFORE:= Sidekiq::Workflow::ClientMiddleware called with:\n worker_class: #{worker_class}\n msg: #{msg}\n queue: #{queue}\n redis_pool: #{redis_pool}"
|
6
|
+
if is_sw_job? msg['args']
|
7
|
+
parse(msg['args']).tap do |params|
|
8
|
+
msg['sw_wid'] = params[1]
|
9
|
+
msg['sw_nid'] = params[2]
|
10
|
+
end
|
11
|
+
end
|
12
|
+
puts "AFTER:= Sidekiq::Workflow::ClientMiddleware called with:\n worker_class: #{worker_class}\n msg: #{msg}\n queue: #{queue}\n redis_pool: #{redis_pool}"
|
13
|
+
yield
|
14
|
+
end
|
15
|
+
|
16
|
+
protected
|
17
|
+
|
18
|
+
def is_sw_job?(args)
|
19
|
+
args.is_a?(Array) &&
|
20
|
+
args.last.is_a?(Array) &&
|
21
|
+
args.last.include?('sidekiq-workflow')
|
22
|
+
end
|
23
|
+
|
24
|
+
def parse(args)
|
25
|
+
# This also strips the sidekiq-workflow params from msg['args']
|
26
|
+
@args ||= args.pop
|
27
|
+
end
|
28
|
+
|
29
|
+
def set_queued_ids(jid)
|
30
|
+
Workflow.find(wid)
|
31
|
+
end
|
32
|
+
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
module Sidekiq
|
2
|
+
module Workflow
|
3
|
+
class Manager
|
4
|
+
attr_accessor :definition
|
5
|
+
|
6
|
+
def self.parse(workflow_definition)
|
7
|
+
[].tap do |h|
|
8
|
+
workflow_definition.each_with_index do |node, i|
|
9
|
+
if node.is_a? Array
|
10
|
+
h[i] = self.parse(node)
|
11
|
+
else
|
12
|
+
h[i] = Node.create({
|
13
|
+
name: node,
|
14
|
+
description: '',
|
15
|
+
params: [],
|
16
|
+
queued_ids: [],
|
17
|
+
completed_ids: []
|
18
|
+
}).id
|
19
|
+
|
20
|
+
end
|
21
|
+
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
def self.execute(workflow)
|
27
|
+
n = workflow.nodes.first
|
28
|
+
|
29
|
+
if n.is_a? Array
|
30
|
+
n.map do |id|
|
31
|
+
node = Node.get(id)
|
32
|
+
class_eval "#{node.name}.perform_async(['sidekiq-workflow', #{workflow.id}, #{node.id}])"
|
33
|
+
end
|
34
|
+
else
|
35
|
+
node = Node.get(n)
|
36
|
+
class_eval "#{node.name}.perform_async(['sidekiq-workflow', #{workflow.id}, #{node.id}])"
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
@@ -0,0 +1,14 @@
|
|
1
|
+
class Sidekiq::Workflow::Node
|
2
|
+
include DataMapper::Resource
|
3
|
+
|
4
|
+
property :id, Serial
|
5
|
+
property :name, String
|
6
|
+
property :description, String
|
7
|
+
|
8
|
+
property :params, Json
|
9
|
+
property :completed_ids, Json
|
10
|
+
property :queued_ids, Json
|
11
|
+
|
12
|
+
end
|
13
|
+
|
14
|
+
DataMapper.finalize
|
File without changes
|
@@ -0,0 +1,23 @@
|
|
1
|
+
module Sidekiq
|
2
|
+
module Workflow
|
3
|
+
|
4
|
+
class ServerMiddleware
|
5
|
+
def call(worker_class, msg, queue)
|
6
|
+
puts "Sidekiq::Workflow::ServerMiddleware called with:\n worker_class: #{worker_class}\n msg: #{msg}\n queue: #{queue}"
|
7
|
+
|
8
|
+
yield
|
9
|
+
# find wid, find nid within the workflow, then move jid from queued_ids to completed_ids
|
10
|
+
if msg['sw_wid'] && msg['sw_nid']
|
11
|
+
workflow = Sidekiq::Workflow::Workflow.get(msg['sw_wid'])
|
12
|
+
nodes = workflow.nodes.map { |n| Node.get(n) }
|
13
|
+
|
14
|
+
workflow.nodes.shift
|
15
|
+
if workflow.save
|
16
|
+
Manager.execute(workflow) unless workflow.nodes.empty?
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
data/sidekiq-workflow.gemspec
CHANGED
@@ -19,9 +19,13 @@ Gem::Specification.new do |spec|
|
|
19
19
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
20
20
|
spec.require_paths = ["lib"]
|
21
21
|
|
22
|
-
spec.
|
22
|
+
spec.required_ruby_version = '1.9'
|
23
|
+
spec.add_development_dependency "bundler", "~> 1.7"
|
23
24
|
spec.add_development_dependency "rake", "~> 10.0"
|
24
25
|
spec.add_development_dependency "rspec", "~> 3.2"
|
26
|
+
spec.add_development_dependency "pry"
|
25
27
|
|
26
28
|
spec.add_runtime_dependency "sidekiq", "~> 3.3"
|
29
|
+
spec.add_runtime_dependency "dm-redis-adapter"
|
30
|
+
spec.add_runtime_dependency "dm-types"
|
27
31
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-workflow
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Michael Kelly
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-08-09 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -16,14 +16,14 @@ dependencies:
|
|
16
16
|
requirements:
|
17
17
|
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version: '1.
|
19
|
+
version: '1.7'
|
20
20
|
type: :development
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version: '1.
|
26
|
+
version: '1.7'
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: rake
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -52,6 +52,20 @@ dependencies:
|
|
52
52
|
- - "~>"
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '3.2'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: pry
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
name: sidekiq
|
57
71
|
requirement: !ruby/object:Gem::Requirement
|
@@ -66,6 +80,34 @@ dependencies:
|
|
66
80
|
- - "~>"
|
67
81
|
- !ruby/object:Gem::Version
|
68
82
|
version: '3.3'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: dm-redis-adapter
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ">="
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
type: :runtime
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: dm-types
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '0'
|
104
|
+
type: :runtime
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - ">="
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '0'
|
69
111
|
description: Allows the execution of mixed parallel and sequential workflows. A workflow
|
70
112
|
can be defined at runtime and executed with full sidekiq integration.
|
71
113
|
email:
|
@@ -84,8 +126,14 @@ files:
|
|
84
126
|
- Rakefile
|
85
127
|
- bin/console
|
86
128
|
- bin/setup
|
129
|
+
- dump.rdb
|
87
130
|
- lib/sidekiq/workflow.rb
|
131
|
+
- lib/sidekiq/workflow/client_middleware.rb
|
132
|
+
- lib/sidekiq/workflow/manager.rb
|
133
|
+
- lib/sidekiq/workflow/monitor.rb
|
134
|
+
- lib/sidekiq/workflow/node.rb
|
88
135
|
- lib/sidekiq/workflow/railtie.rb
|
136
|
+
- lib/sidekiq/workflow/server_middleware.rb
|
89
137
|
- lib/sidekiq/workflow/version.rb
|
90
138
|
- sidekiq-workflow.gemspec
|
91
139
|
homepage: https://github.com/michaelkelly322/sidekiq-workflow
|
@@ -98,9 +146,9 @@ require_paths:
|
|
98
146
|
- lib
|
99
147
|
required_ruby_version: !ruby/object:Gem::Requirement
|
100
148
|
requirements:
|
101
|
-
- -
|
149
|
+
- - '='
|
102
150
|
- !ruby/object:Gem::Version
|
103
|
-
version: '
|
151
|
+
version: '1.9'
|
104
152
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
105
153
|
requirements:
|
106
154
|
- - ">="
|
@@ -108,7 +156,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
108
156
|
version: '0'
|
109
157
|
requirements: []
|
110
158
|
rubyforge_project:
|
111
|
-
rubygems_version: 2.4.
|
159
|
+
rubygems_version: 2.4.6
|
112
160
|
signing_key:
|
113
161
|
specification_version: 4
|
114
162
|
summary: Complex workflow management in Sidekiq
|