parallel_workforce 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
data/LICENSE ADDED
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
data/README.md ADDED
@@ -0,0 +1,110 @@
1
+ # ParallelWorkforce
2
+
3
+ Easily parallelize functionality by partitioning work in either a Sidekiq (preferred) or ActiveJob worker pool.
4
+
5
+ See more info at the EnjoyTech blog.
6
+
7
+ ## Installation
8
+
9
+ Add this line to your application's Gemfile:
10
+
11
+ ```ruby
12
+ gem 'parallel_workforce'
13
+ ```
14
+
15
+ And then execute:
16
+
17
+ $ bundle
18
+
19
+ Or install it yourself as:
20
+
21
+ $ gem install parallel_workforce
22
+
23
+ ## Usage
24
+
25
+ Parallel execution requires two things:
26
+
27
+ * Creating one or more actor classes that has a keyword arguments `initializer` and a no-args `perform` method.
28
+ * Invoking `ParallelWorkforce.perform_all` with an array of `actor_classes` and an equally sized array for `actor_args_array`
29
+ with initializer arguments for the actor classes.
30
+
31
+ Optional arguments for `execute_serially` and `job_class` can be used with `perform_all` to override configuration settings.
32
+
33
+ An optional `serial_execution_indexes` can also be used to execute specific indexes serially. This is useful to easily
34
+ perform "fast" actors in the current thread and only sending "slow" actors to be executed in workers. The specified serial
35
+ actors are performed after enqueuing the remaining actors and waiting for them to complete.
36
+
37
+ Additionally, a block can be passed to `perform_all` that allows arbitrary
38
+ execution in the current thread after the jobs have been enqueued and before checking for job responses.
39
+
40
+ ## Example
41
+
42
+ Create cryptographic hashes of passwords in parallel in workers using BCrypt. Then verify the generated hashes against the passwords.
43
+
44
+ ### Create a class that performs the work.
45
+
46
+ ```ruby
47
+ require 'bcrypt'
48
+
49
+ class PasswordHashGenerator
50
+ attr_reader :password
51
+
52
+ def initialize(password:)
53
+ @password = password
54
+ end
55
+
56
+ def perform
57
+ BCrypt::Password.create(password, cost: 15).to_s
58
+ end
59
+ end
60
+ ```
61
+
62
+ ### Invoke the workers and verify the results.
63
+
64
+ ```ruby
65
+ require 'bcrypt'
66
+
67
+ passwords = ['password 1', 'password 2', 'password 3']
68
+
69
+ password_hashes = ParallelWorkforce.perform_all(
70
+ actor_classes: Array.new(passwords.size) { PasswordHashGenerator },
71
+ actor_args_array: passwords.map { |password| { password: password } },
72
+ )
73
+
74
+ # check that password hashes match
75
+ passwords.zip(password_hashes).each do |password, password_hash|
76
+ raise "Password does not match hash" if BCrypt::Password.new(password_hash) != password
77
+ end
78
+ ```
79
+
80
+ ## Configuration
81
+
82
+ Execute `ParallelWorkforce.configure` before executing `ParallelWorkforce.perform_all` to configure. In a Rails application, add a file `parallel_workforce_config.rb` in your `config/initializers` directory.
83
+
84
+ ```ruby
85
+ ParallelWorkforce.configure do |configuration|
86
+ configuration.job_class = MyJobClass
87
+ # etc
88
+ end
89
+ ```
90
+
91
+ * `job_class` - The class to use to respond to enqueued jobs. See classes under the `ParallelWorkforce::Job` namespace. Defaults to `ParallelWorkforce::Job::ActiveJob`.
92
+ * `logger` - Defaults to `Rails.logger` if `Rails` is loaded.
93
+ * `revision_builder` - Used to determine if the calling thread and worker have a different revision of code. Defaults to a class that builds a hash from the contents of all `.rb` files contained within the current working directory.
94
+ * `serial_mode_checker` - An object with an `execute_serially?` method that allows forcing Actors to execute in calling thread instead of in workers. This can be used to turn off parallel execution globally. Default is `nil`.
95
+ * `serializer` - An object with a `serialize(object)` method that returns a `String` and `deserialize(string)` method that returns an `Object`. Default imlementation uses `Marshal.dump` and `Marshal.load`.
96
+ * `redis_connector` - An object with a `with` method that yields a Redis connection. Defaults to `ParallelWorkforce::RedisConnector::RedisPool`. If using Sidekiq, it's best to use `ParallelWorkforce::RedisConnector::SidekiqRedisPool`.
97
+ * `job_timeout` - Time allowed to execte a job before timing out. Default is `10` seconds.
98
+ * `job_key_expiration` - Time allowed for result key in Redis to remain before it expires. This should be larger than `job_timeout`. Default is `20` seconds.
99
+ * `production_environment` - Removes serialization/deserialization when executing serially that helps locate problems with objects that fail to serialize. When `Rails` is loaded, uses `Rails.env.production?`, otherwise `true`.
100
+ * `allow_nested_parallelization` - By default, executing `ParallelWorkforce.perform_all` within a ParallelWorkforce `Actor` will execute serially. This can be enabled to allow nesting worker invocations, but ensure that the worker pool is large enough to handle blocked workers waiting for a response. Too small of a pool will lead to timeouts as no workers will be available.
101
+
102
+ ## Development
103
+
104
+ After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
105
+
106
+ To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
107
+
108
+ ## Contributing
109
+
110
+ Bug reports and pull requests are welcome on GitHub at https://github.com/EnjoyTech/parallel_workforce.
data/Rakefile ADDED
@@ -0,0 +1,2 @@
1
+ require "bundler/gem_tasks"
2
+ task default: :spec
data/bin/console ADDED
@@ -0,0 +1,14 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require "bundler/setup"
4
+ require "parallel_workforce"
5
+
6
+ # You can add fixtures and/or initialization code here to make experimenting
7
+ # with your gem easier. You can also use a different console, if you like.
8
+
9
+ # (If you use this, don't forget to add pry to your Gemfile!)
10
+ # require "pry"
11
+ # Pry.start
12
+
13
+ require "irb"
14
+ IRB.start(__FILE__)
data/bin/setup ADDED
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+ IFS=$'\n\t'
4
+ set -vx
5
+
6
+ bundle install
7
+
8
+ # Do any other automated setup that you need to do here
@@ -0,0 +1,29 @@
1
+ module ParallelWorkforce
2
+ class Configuration
3
+ attr_accessor(
4
+ :job_class,
5
+ :logger,
6
+ :revision_builder,
7
+ :serial_mode_checker,
8
+ :serializer,
9
+ :redis_connector,
10
+ :job_timeout,
11
+ :job_key_expiration,
12
+ :production_environment,
13
+ :allow_nested_parallelization,
14
+ )
15
+
16
+ def initialize
17
+ @job_class = defined?(ActiveJob) && ParallelWorkforce::Job::ActiveJob
18
+ @logger = defined?(Rails) && Rails.logger
19
+ @revision_builder = ParallelWorkforce::RevisionBuilder::FilesHash.new
20
+ @serial_mode_checker = nil
21
+ @serializer = ParallelWorkforce::Serializer::Marshal.new
22
+ @redis_connector = ParallelWorkforce::RedisConnector::RedisPool.new
23
+ @job_timeout = 10
24
+ @job_key_expiration = 20
25
+ @production_environment = defined?(Rails) ? Rails.env.production? : true
26
+ @allow_nested_parallelization = false
27
+ end
28
+ end
29
+ end
@@ -0,0 +1,224 @@
1
+ require 'securerandom'
2
+
3
+ module ParallelWorkforce
4
+ # rubocop:disable Metrics/ClassLength
5
+ class Executor
6
+ attr_reader(
7
+ :actor_classes,
8
+ :actor_args_array,
9
+ :execute_serially,
10
+ :serial_execution_indexes,
11
+ :job_class,
12
+ :execution_block,
13
+ )
14
+
15
+ # +actor_classes+: a single class or array of classes that can be instantiated with no args and have a `perform` method.
16
+ # If an array is passed, the array size must batch the actor_args_array size.
17
+ # Return results array with element from each actor in the order of the actor_args_array
18
+ # rubocop:disable Metrics/ParameterLists
19
+ def initialize(actor_classes:, actor_args_array:,
20
+ serial_execution_indexes: nil, execute_serially: nil, job_class: nil, execution_block: nil)
21
+ @actor_classes = normalize_actor_classes!(actor_classes, actor_args_array)
22
+ @actor_args_array = actor_args_array
23
+ @serial_execution_indexes = serial_execution_indexes
24
+ @execute_serially = execute_serially.nil? ? calculate_execute_serially : execute_serially
25
+ @job_class = job_class || configuration.job_class
26
+ @execution_block = execution_block
27
+ end
28
+ # rubocop:enable Metrics/ParameterLists
29
+
30
+ def perform_all
31
+ serial_execution_indexes = calculate_serial_execution_indexes
32
+ serial_actor_classes, serial_actor_args_array, parallel_actor_classes, parallel_actor_args_array =
33
+ split_serial_parallel(serial_execution_indexes, actor_classes, actor_args_array)
34
+
35
+ serial_results = nil
36
+ parallel_results = execute_actors(parallel_actor_classes, parallel_actor_args_array) do
37
+ serial_results = serial_actor_classes.zip(serial_actor_args_array).collect do |actor_class, actor_args|
38
+ execute_actor_serially(actor_class, actor_args)
39
+ end
40
+ end
41
+
42
+ Array.new(actor_args_array.length) do |index|
43
+ if index == serial_execution_indexes.first
44
+ serial_execution_indexes.shift
45
+ serial_results.shift
46
+ else
47
+ parallel_results.shift
48
+ end
49
+ end
50
+ end
51
+
52
+ private
53
+
54
+ def split_serial_parallel(serial_execution_indexes, actor_classes, actor_args_array)
55
+ serial_execution_indexes = serial_execution_indexes.dup
56
+
57
+ [[], [], [], []].tap do |result|
58
+ actor_classes.zip(actor_args_array).each_with_index do |(actor_class, actor_args), index|
59
+ if index == serial_execution_indexes.first
60
+ serial_execution_indexes.shift
61
+ result[0] << actor_class
62
+ result[1] << actor_args
63
+ else
64
+ result[2] << actor_class
65
+ result[3] << actor_args
66
+ end
67
+ end
68
+ end
69
+ end
70
+
71
+ def calculate_serial_execution_indexes
72
+ if execute_serially
73
+ actor_args_array.length.times.to_a
74
+ elsif serial_execution_indexes
75
+ serial_execution_indexes.sort.each do |index|
76
+ if index < 0 || index >= actor_args_array.size
77
+ raise ArgumentError.new("serial_execution_indexes must be between 0 and #{actor_args_array.size}")
78
+ end
79
+ end
80
+ else
81
+ []
82
+ end
83
+ end
84
+
85
+ def configuration
86
+ ParallelWorkforce.configuration
87
+ end
88
+
89
+ def calculate_execute_serially
90
+ if ParallelWorkforce::Job::Util::Performer.parallel_workforce_thread? && !configuration.allow_nested_parallelization
91
+ return true
92
+ end
93
+
94
+ configuration.serial_mode_checker&.execute_serially?
95
+ end
96
+
97
+ def serialize(object)
98
+ configuration.serializer.serialize(object)
99
+ end
100
+
101
+ def deserialize(string)
102
+ configuration.serializer.deserialize(string)
103
+ end
104
+
105
+ def serialize_and_deserialize(object)
106
+ return object if configuration.production_environment
107
+
108
+ deserialize(serialize(object))
109
+ end
110
+
111
+ def normalize_actor_classes!(actor_classes, actor_args_array)
112
+ actor_classes = actor_classes.is_a?(Array) ? actor_classes : Array.new(actor_args_array.length) { actor_classes }
113
+
114
+ if actor_classes.length != actor_args_array.length
115
+ raise ArgumentError.new("actor_classes and actor_args_array must be same length")
116
+ end
117
+
118
+ actor_classes
119
+ end
120
+
121
+ # rubocop:disable Metrics/AbcSize
122
+ def execute_actors(actor_classes, actor_args_array, &execute_actors_serially_proc)
123
+ result_key = "ParallelWorkforce::Executor:result_key:#{SecureRandom.uuid}"
124
+
125
+ result = []
126
+
127
+ actor_classes.zip(actor_args_array).each.with_index do |(actor_class, actor_args), index|
128
+ enqueue_actor(actor_class, actor_args, index, result_key)
129
+ end
130
+
131
+ configuration.redis_connector.with do |redis|
132
+ redis.expire(result_key, configuration.job_key_expiration)
133
+ end
134
+
135
+ execute_actors_serially_proc.call
136
+
137
+ execution_block&.call
138
+
139
+ # concat results from enqueued actors
140
+ result.concat(wait_for_actor_results(result_key, actor_classes, actor_args_array))
141
+
142
+ result
143
+ ensure
144
+ configuration.redis_connector.with do |redis|
145
+ redis.del(result_key) if result_key
146
+ end
147
+ end
148
+ # rubocop:enable Metrics/AbcSize
149
+
150
+ def execute_actor_serially(actor_class, actor_args)
151
+ # Mimic serialization behavior in parallel actor execution
152
+ serialize_and_deserialize(
153
+ ParallelWorkforce::Job::Util::Performer.perform(
154
+ actor_class,
155
+ serialize_and_deserialize(actor_args),
156
+ ),
157
+ )
158
+ end
159
+
160
+ def enqueue_actor(actor_class, actor_args, index, result_key)
161
+ job_class.enqueue_actor(
162
+ actor_class_name: actor_class.name,
163
+ result_key: result_key,
164
+ index: index,
165
+ server_revision: configuration.revision_builder&.revision,
166
+ serialized_actor_args: serialize(actor_args),
167
+ )
168
+ end
169
+
170
+ # rubocop:disable Metrics/MethodLength
171
+ # rubocop:disable Metrics/AbcSize
172
+ def wait_for_actor_results(result_key, actor_classes, actor_args_array)
173
+ return [] if (num_results = actor_classes.length) == 0
174
+
175
+ result_values = Array.new(num_results)
176
+ result_count = 0
177
+
178
+ configuration.redis_connector.with do |redis|
179
+ begin
180
+ Timeout.timeout(configuration.job_timeout) do
181
+ until result_count == num_results
182
+ _key, response = redis.blpop(result_key, configuration.job_timeout)
183
+ raise ParallelWorkforce::TimeoutError.new("Timeout waiting for Redis#blpop") if response.nil?
184
+
185
+ result_count += 1
186
+
187
+ index, value = parse_response!(response, actor_classes, actor_args_array)
188
+
189
+ result_values[index] = value
190
+ end
191
+ end
192
+ rescue Timeout::Error
193
+ raise ParallelWorkforce::TimeoutError.new("Timeout from ParallelWorkforce job")
194
+ end
195
+ end
196
+
197
+ result_values
198
+ end
199
+ # rubocop:enable Metrics/AbcSize
200
+ # rubocop:enable Metrics/MethodLength
201
+
202
+ def parse_response!(response, actor_classes, actor_args_array)
203
+ # NOTE: Always stored in Redis with Ruby marshaling, not serializer
204
+ message = Marshal.load(response)
205
+ if (error = message[:error])
206
+ raise ParallelWorkforce::ActorPerformError.new("Error received from parallel actor: #{error}")
207
+ elsif message[:error_revision]
208
+ # recoverable error
209
+ index = message[:index]
210
+ [index, execute_actor_serially(actor_classes[index], actor_args_array[index])]
211
+ else
212
+ begin
213
+ [message[:index], deserialize(message[:serialized_value])]
214
+ rescue ParallelWorkforce::SerializerError => e
215
+ ParallelWorkforce.log(:warn, "Cannot deserialize serialized_value: #{e}")
216
+
217
+ index = message[:index]
218
+ [index, execute_actor_serially(actor_classes[index], actor_args_array[index])]
219
+ end
220
+ end
221
+ end
222
+ end
223
+ # rubocop:enable Metrics/ClassLength
224
+ end
@@ -0,0 +1,23 @@
1
+ module ParallelWorkforce
2
+ module Job
3
+ class ActiveJob < ActiveJob::Base
4
+ discard_on Exception if defined?(discard_on)
5
+
6
+ class << self
7
+ def enqueue_actor(actor_class_name:, result_key:, index:, server_revision:, serialized_actor_args:)
8
+ perform_later(
9
+ actor_class_name: actor_class_name,
10
+ result_key: result_key,
11
+ index: index,
12
+ server_revision: server_revision,
13
+ serialized_actor_args: serialized_actor_args,
14
+ )
15
+ end
16
+ end
17
+
18
+ def perform(args)
19
+ ParallelWorkforce::Job::Util::Performer.new(**args.symbolize_keys).perform
20
+ end
21
+ end
22
+ end
23
+ end
@@ -0,0 +1,25 @@
1
+ module ParallelWorkforce
2
+ module Job
3
+ class ActiveJobRails < ParallelWorkforce::Job::ActiveJob
4
+ class << self
5
+ def enqueue_actor(actor_class_name:, result_key:, index:, server_revision:, serialized_actor_args:)
6
+ perform_later(
7
+ actor_class_name: actor_class_name,
8
+ result_key: result_key,
9
+ index: index,
10
+ server_revision: server_revision,
11
+ serialized_actor_args: serialized_actor_args,
12
+ time_zone_name: Time.zone.name,
13
+ )
14
+ end
15
+ end
16
+
17
+ def perform(args)
18
+ args.symbolize_keys!
19
+ Time.use_zone(args.delete(:time_zone_name)) do
20
+ ParallelWorkforce::Job::Util::Performer.new(**args).perform
21
+ end
22
+ end
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,26 @@
1
+ module ParallelWorkforce
2
+ module Job
3
+ class Sidekiq
4
+ include ::Sidekiq::Worker
5
+
6
+ sidekiq_options retry: false
7
+
8
+ class << self
9
+ def enqueue_actor(actor_class_name:, result_key:, index:, server_revision:, serialized_actor_args:)
10
+ perform_async(
11
+ actor_class_name: actor_class_name,
12
+ result_key: result_key,
13
+ index: index,
14
+ server_revision: server_revision,
15
+ serialized_actor_args: serialized_actor_args,
16
+ )
17
+ end
18
+ end
19
+
20
+ def perform(args)
21
+ args = args.each_with_object({}) { |(k, v), result| result[k.to_sym] = v }
22
+ ParallelWorkforce::Job::Util::Performer.new(**args).perform
23
+ end
24
+ end
25
+ end
26
+ end