sidekiq-workflow 0.0.1 → 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: fe0a36c7bd5d7ddf8d45311228f214d85776082a
4
- data.tar.gz: d4313404aff1c62630a60c6a21a7e6fa0ae415ed
3
+ metadata.gz: c896e5dd3c9e174a591c2485eef0c2f8f993847f
4
+ data.tar.gz: d8d07782138963a6773a9ac6491316173a067aaa
5
5
  SHA512:
6
- metadata.gz: 48b8abdbfb49c14a60a0d8ce19817a092c5558f51fe1ed86cc5ed74bd0b86ef4336e313d9f0ade78177c0a57f409c765757d7788e1cebc5f495f3bef988d29d5
7
- data.tar.gz: ffb16abb61d2367583a06be9bc0d7341706e0e1dda1ecee7e74f55adbfb025ebd92bbf5d19218cc7ee298d39636658e768653949c4a1549baa705fbfd3ee4fb9
6
+ metadata.gz: ac3613923dca6a2881b4a75a3eae57adcaf118e88584af49fd1b24dacb7c7d991f2afc0d504396131cac1a03b76e4ebd32f8ca073d7ea7fead948171fa8470ee
7
+ data.tar.gz: 3a4f57e72957c252ff73f9f369caed3d9e53c11957769bd4c51b1bdfef498cd2c5fbe3115d975621c0e5d3d9e0896fbb6b21959cbaf503d347d1f3b3c8adb617
data/.gitignore CHANGED
@@ -8,3 +8,4 @@
8
8
  /spec/reports/
9
9
  /tmp/
10
10
  *.gem
11
+ .DS_Store
File without changes
data/Gemfile CHANGED
File without changes
File without changes
data/README.md CHANGED
@@ -1,4 +1,7 @@
1
1
  # Sidekiq::Workflow
2
+ _sidekiq-workflow is currently in development and is not yet suited to
3
+ production use. Please continue at your own risk...YOU HAVE BEEN
4
+ WARNED!_
2
5
 
3
6
  ## Installation
4
7
 
@@ -18,7 +21,72 @@ Or install it yourself as:
18
21
 
19
22
  ## Usage
20
23
 
21
- TODO: Write usage instructions here
24
+ To run a workflow, sidekiq-workflow provides a simple interface:
25
+
26
+ ```
27
+ Sidekiq::Workflow.run(sequence_definition)
28
+ ```
29
+
30
+ A sequence definition is composed of an array-defined workflow. The
31
+ convention here is that jobs within the root array element are executed
32
+ sequentially, one level deeper is executed in parallel. Currently,
33
+ sidekiq-workflow only supports simple embedded parallelism at one level.
34
+ This means that it only supports workflows like the following:
35
+
36
+ ```
37
+ worker_2
38
+ / \
39
+ worker_1 worker_4
40
+ \ /
41
+ worker_3
42
+ ```
43
+ A workflow like this is easily defined with the following structure:
44
+ ```ruby
45
+ [
46
+ worker_1,
47
+ [worker_2, worker_3],
48
+ worker_4
49
+ ]
50
+ ```
51
+ workers in the root array are processed sequentially and any workers in nested
52
+ arrays are processes in parrallel. The `worker_#` values above can be a
53
+ strings or worker class names so:
54
+
55
+ ```ruby
56
+ [
57
+ "AccountWorker",
58
+ ["UserWorker", "ProfileWorker"],
59
+ "EmailWorker"
60
+ ]
61
+ ```
62
+
63
+ is equivalent to
64
+
65
+ ```ruby
66
+ [
67
+ AccountWorker,
68
+ [UserWorker, ProfileWorker],
69
+ EmailWorker
70
+ ]
71
+ ```
72
+
73
+ ## Coming Soon
74
+
75
+ Parameter passing and custom worker methods will be coming shortly as
76
+ well as expanded workflow structure providing for any mixture of
77
+ sequential and parallel definitions to make workflows like this
78
+ possible:
79
+
80
+ ```
81
+ worker_5
82
+ / \
83
+ worker_4 ----worker_7
84
+ / \ / /
85
+ worker_1 worker_6 /
86
+ \ /
87
+ worker_2---worker_3
88
+
89
+ ```
22
90
 
23
91
  ## Development
24
92
 
@@ -30,7 +98,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
30
98
 
31
99
  ## Contributing
32
100
 
33
- Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/sidekiq-workflow. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct.
101
+ Bug reports and pull requests are welcome on GitHub at https://github.com/michaelkelly322/sidekiq-workflow. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
34
102
 
35
103
  Please also take a look at [this post](https://badmonkeydev.wordpress.com/2015/05/23/code-review-best-practices/) for guidlines on submitting pull requests. Thanks!
36
104
 
data/Rakefile CHANGED
File without changes
Binary file
@@ -1,11 +1,49 @@
1
1
  require "sidekiq"
2
+ require "redis-namespace"
3
+ require "dm-core"
4
+ require "dm-types"
5
+ require "dm-redis-adapter"
2
6
  require "sidekiq/workflow/version"
7
+ require "sidekiq/workflow/client_middleware"
8
+ require "sidekiq/workflow/server_middleware"
9
+ require "sidekiq/workflow/manager"
10
+ require "sidekiq/workflow/node"
3
11
 
4
- module Sidekiq
5
- module Workflow
6
- def self.run(workflow_definition)
7
- #WorkflowValidator.validate workflow_definition
8
- #WorkflowManager.execute workflow_definition
12
+ class TestWorker
13
+ include Sidekiq::Worker
14
+
15
+ def perform
16
+ sleep(5)
17
+ end
18
+ end
19
+
20
+ DataMapper.setup(:default, {:adapter => "redis"})
21
+
22
+ module Sidekiq::Workflow
23
+
24
+ def self.run(workflow_definition)
25
+ workflow = Workflow.create(workflow_definition)
26
+
27
+ Manager.execute workflow
28
+ workflow.id
29
+ end
30
+
31
+ class Workflow
32
+ include DataMapper::Resource
33
+
34
+ property :id, Serial
35
+ property :workflow_definition, Json
36
+ property :nodes, Json
37
+
38
+ def initialize(workflow_definition)
39
+ raise ArgumentError.new('workflow_definition can\'t be nil') unless workflow_definition
40
+ self.workflow_definition = workflow_definition
41
+ self.nodes = parsed_definition
42
+ binding.pry
43
+ end
44
+
45
+ def parsed_definition
46
+ @parsed_definition ||= Manager.parse workflow_definition
9
47
  end
10
48
  end
11
49
  end
@@ -0,0 +1,32 @@
1
+ class Sidekiq::Workflow::ClientMiddleware
2
+ attr_accessor :wid, :nid
3
+
4
+ def call(worker_class, msg, queue, redis_pool)
5
+ puts "BEFORE:= Sidekiq::Workflow::ClientMiddleware called with:\n worker_class: #{worker_class}\n msg: #{msg}\n queue: #{queue}\n redis_pool: #{redis_pool}"
6
+ if is_sw_job? msg['args']
7
+ parse(msg['args']).tap do |params|
8
+ msg['sw_wid'] = params[1]
9
+ msg['sw_nid'] = params[2]
10
+ end
11
+ end
12
+ puts "AFTER:= Sidekiq::Workflow::ClientMiddleware called with:\n worker_class: #{worker_class}\n msg: #{msg}\n queue: #{queue}\n redis_pool: #{redis_pool}"
13
+ yield
14
+ end
15
+
16
+ protected
17
+
18
+ def is_sw_job?(args)
19
+ args.is_a?(Array) &&
20
+ args.last.is_a?(Array) &&
21
+ args.last.include?('sidekiq-workflow')
22
+ end
23
+
24
+ def parse(args)
25
+ # This also strips the sidekiq-workflow params from msg['args']
26
+ @args ||= args.pop
27
+ end
28
+
29
+ def set_queued_ids(jid)
30
+ Workflow.find(wid)
31
+ end
32
+ end
@@ -0,0 +1,41 @@
1
+ module Sidekiq
2
+ module Workflow
3
+ class Manager
4
+ attr_accessor :definition
5
+
6
+ def self.parse(workflow_definition)
7
+ [].tap do |h|
8
+ workflow_definition.each_with_index do |node, i|
9
+ if node.is_a? Array
10
+ h[i] = self.parse(node)
11
+ else
12
+ h[i] = Node.create({
13
+ name: node,
14
+ description: '',
15
+ params: [],
16
+ queued_ids: [],
17
+ completed_ids: []
18
+ }).id
19
+
20
+ end
21
+
22
+ end
23
+ end
24
+ end
25
+
26
+ def self.execute(workflow)
27
+ n = workflow.nodes.first
28
+
29
+ if n.is_a? Array
30
+ n.map do |id|
31
+ node = Node.get(id)
32
+ class_eval "#{node.name}.perform_async(['sidekiq-workflow', #{workflow.id}, #{node.id}])"
33
+ end
34
+ else
35
+ node = Node.get(n)
36
+ class_eval "#{node.name}.perform_async(['sidekiq-workflow', #{workflow.id}, #{node.id}])"
37
+ end
38
+ end
39
+ end
40
+ end
41
+ end
@@ -0,0 +1,11 @@
1
+ class Sidekiq::Workflow::Monitor
2
+ attr_accessor :wid
3
+ def self.[](wid)
4
+ self.new(wid)
5
+ end
6
+
7
+ def initialize(wid)
8
+ self.wid = wid
9
+ end
10
+
11
+ end
@@ -0,0 +1,14 @@
1
+ class Sidekiq::Workflow::Node
2
+ include DataMapper::Resource
3
+
4
+ property :id, Serial
5
+ property :name, String
6
+ property :description, String
7
+
8
+ property :params, Json
9
+ property :completed_ids, Json
10
+ property :queued_ids, Json
11
+
12
+ end
13
+
14
+ DataMapper.finalize
File without changes
@@ -0,0 +1,23 @@
1
+ module Sidekiq
2
+ module Workflow
3
+
4
+ class ServerMiddleware
5
+ def call(worker_class, msg, queue)
6
+ puts "Sidekiq::Workflow::ServerMiddleware called with:\n worker_class: #{worker_class}\n msg: #{msg}\n queue: #{queue}"
7
+
8
+ yield
9
+ # find wid, find nid within the workflow, then move jid from queued_ids to completed_ids
10
+ if msg['sw_wid'] && msg['sw_nid']
11
+ workflow = Sidekiq::Workflow::Workflow.get(msg['sw_wid'])
12
+ nodes = workflow.nodes.map { |n| Node.get(n) }
13
+
14
+ workflow.nodes.shift
15
+ if workflow.save
16
+ Manager.execute(workflow) unless workflow.nodes.empty?
17
+ end
18
+ end
19
+ end
20
+ end
21
+ end
22
+ end
23
+
@@ -1,5 +1,5 @@
1
1
  module Sidekiq
2
2
  module Workflow
3
- VERSION = "0.0.1"
3
+ VERSION = "0.0.2"
4
4
  end
5
5
  end
@@ -19,9 +19,13 @@ Gem::Specification.new do |spec|
19
19
  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
20
20
  spec.require_paths = ["lib"]
21
21
 
22
- spec.add_development_dependency "bundler", "~> 1.10"
22
+ spec.required_ruby_version = '1.9'
23
+ spec.add_development_dependency "bundler", "~> 1.7"
23
24
  spec.add_development_dependency "rake", "~> 10.0"
24
25
  spec.add_development_dependency "rspec", "~> 3.2"
26
+ spec.add_development_dependency "pry"
25
27
 
26
28
  spec.add_runtime_dependency "sidekiq", "~> 3.3"
29
+ spec.add_runtime_dependency "dm-redis-adapter"
30
+ spec.add_runtime_dependency "dm-types"
27
31
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sidekiq-workflow
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Michael Kelly
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2015-06-09 00:00:00.000000000 Z
11
+ date: 2015-08-09 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -16,14 +16,14 @@ dependencies:
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: '1.10'
19
+ version: '1.7'
20
20
  type: :development
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: '1.10'
26
+ version: '1.7'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: rake
29
29
  requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,20 @@ dependencies:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
54
  version: '3.2'
55
+ - !ruby/object:Gem::Dependency
56
+ name: pry
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
55
69
  - !ruby/object:Gem::Dependency
56
70
  name: sidekiq
57
71
  requirement: !ruby/object:Gem::Requirement
@@ -66,6 +80,34 @@ dependencies:
66
80
  - - "~>"
67
81
  - !ruby/object:Gem::Version
68
82
  version: '3.3'
83
+ - !ruby/object:Gem::Dependency
84
+ name: dm-redis-adapter
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - ">="
88
+ - !ruby/object:Gem::Version
89
+ version: '0'
90
+ type: :runtime
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - ">="
95
+ - !ruby/object:Gem::Version
96
+ version: '0'
97
+ - !ruby/object:Gem::Dependency
98
+ name: dm-types
99
+ requirement: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ type: :runtime
105
+ prerelease: false
106
+ version_requirements: !ruby/object:Gem::Requirement
107
+ requirements:
108
+ - - ">="
109
+ - !ruby/object:Gem::Version
110
+ version: '0'
69
111
  description: Allows the execution of mixed parallel and sequential workflows. A workflow
70
112
  can be defined at runtime and executed with full sidekiq integration.
71
113
  email:
@@ -84,8 +126,14 @@ files:
84
126
  - Rakefile
85
127
  - bin/console
86
128
  - bin/setup
129
+ - dump.rdb
87
130
  - lib/sidekiq/workflow.rb
131
+ - lib/sidekiq/workflow/client_middleware.rb
132
+ - lib/sidekiq/workflow/manager.rb
133
+ - lib/sidekiq/workflow/monitor.rb
134
+ - lib/sidekiq/workflow/node.rb
88
135
  - lib/sidekiq/workflow/railtie.rb
136
+ - lib/sidekiq/workflow/server_middleware.rb
89
137
  - lib/sidekiq/workflow/version.rb
90
138
  - sidekiq-workflow.gemspec
91
139
  homepage: https://github.com/michaelkelly322/sidekiq-workflow
@@ -98,9 +146,9 @@ require_paths:
98
146
  - lib
99
147
  required_ruby_version: !ruby/object:Gem::Requirement
100
148
  requirements:
101
- - - ">="
149
+ - - '='
102
150
  - !ruby/object:Gem::Version
103
- version: '0'
151
+ version: '1.9'
104
152
  required_rubygems_version: !ruby/object:Gem::Requirement
105
153
  requirements:
106
154
  - - ">="
@@ -108,7 +156,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
108
156
  version: '0'
109
157
  requirements: []
110
158
  rubyforge_project:
111
- rubygems_version: 2.4.7
159
+ rubygems_version: 2.4.6
112
160
  signing_key:
113
161
  specification_version: 4
114
162
  summary: Complex workflow management in Sidekiq