acidic_job 1.0.0.pre9 → 1.0.0.pre12
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.tool-versions +1 -0
- data/Gemfile.lock +1 -1
- data/README.md +49 -0
- data/lib/acidic_job/extensions/action_mailer.rb +6 -4
- data/lib/acidic_job/extensions/active_job.rb +2 -1
- data/lib/acidic_job/extensions/noticed.rb +3 -1
- data/lib/acidic_job/extensions/sidekiq.rb +4 -2
- data/lib/acidic_job/idempotency_key.rb +56 -6
- data/lib/acidic_job/staging.rb +10 -0
- data/lib/acidic_job/version.rb +1 -1
- data/lib/acidic_job.rb +40 -26
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1af7bca23aa5ead8fed158fb73565736555236bf4515ab73f631a3013e138f12
|
4
|
+
data.tar.gz: cc1118240efe70d71dcd7c2eb07435f3e56934c2c3e8653848a9c3467dc60b83
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 90073b7d2964d65886ca2fd4320481455e04218db92ab3b2ccd65bdffc527b03f8c928c58590ed62cf7734c1702687978dd5f1406ea740f13f1fbccb7c27b48d
|
7
|
+
data.tar.gz: 9f2910c1620f489a1bb5a310821ead5363671aa0ffc9a21f909894f43c2de52b056ee096f17455fb8b34bd55d8f521ca71760df3959b8b0c1f90562c50129ff4
|
data/.tool-versions
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
ruby 3.0.2
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,8 @@
|
|
1
1
|
# AcidicJob
|
2
2
|
|
3
|
+
[![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://badge.fury.io/rb/acidic_job)
|
4
|
+
![main workflow](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
|
5
|
+
|
3
6
|
### Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
|
4
7
|
|
5
8
|
At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
|
@@ -64,6 +67,7 @@ It provides a suite of functionality that empowers you to create complex, robust
|
|
64
67
|
* Transactional Steps — break your job into a series of steps, each of which will be run within an acidic database transaction, allowing retries to jump back to the last "recovery point".
|
65
68
|
* Persisted Attributes — when retrying jobs at later steps, we need to ensure that data created in previous steps is still available to later steps on retry.
|
66
69
|
* Transactionally Staged Jobs — enqueue additional jobs within the acidic transaction safely
|
70
|
+
* Custom Idempotency Keys — use something other than the job ID for the idempotency key of the job run
|
67
71
|
* Sidekiq Callbacks — bring ActiveJob-like callbacks into your pure Sidekiq Workers
|
68
72
|
* Sidekiq Batches — leverage the power of Sidekiq Pro's `batch` functionality without the hassle
|
69
73
|
|
@@ -163,6 +167,51 @@ class RideCreateJob < ActiveJob::Base
|
|
163
167
|
end
|
164
168
|
```
|
165
169
|
|
170
|
+
### Custom Idempotency Keys
|
171
|
+
|
172
|
+
By default, `AcidicJob` uses the job identifier provided by the queueing system (ActiveJob or Sidekiq) as the idempotency key for the job run. The idempotency key is what is used to guarantee that no two runs of the same job occur. However, sometimes we need particular jobs to be idempotent based on some other criteria. In these cases, `AcidicJob` provides a collection of tools to allow you to ensure the idempotency of your jobs.
|
173
|
+
|
174
|
+
Firstly, you can configure your job class to explicitly use either the job identifier or the job arguments as the foundation for the idempotency key. A job class that calls the `acidic_by_job_id` class method (which is the default behavior) will simply make the job run's idempotency key the job's identifier:
|
175
|
+
|
176
|
+
```ruby
|
177
|
+
class ExampleJob < ActiveJob::Base
|
178
|
+
include AcidicJob
|
179
|
+
acidic_by_job_id
|
180
|
+
|
181
|
+
def perform
|
182
|
+
end
|
183
|
+
end
|
184
|
+
```
|
185
|
+
|
186
|
+
Conversely, a job class can use the `acidic_by_job_args` method to configure that job class to use the arguments passed to the job as the foundation for the job run's idempotency key:
|
187
|
+
|
188
|
+
```ruby
|
189
|
+
class ExampleJob < ActiveJob::Base
|
190
|
+
include AcidicJob
|
191
|
+
acidic_by_job_args
|
192
|
+
|
193
|
+
def perform(arg_1, arg_2)
|
194
|
+
# the idempotency key will be based on whatever the values of `arg_1` and `arg_2` are
|
195
|
+
end
|
196
|
+
end
|
197
|
+
```
|
198
|
+
|
199
|
+
These options cover the two common situations, but sometimes our systems need finer-grained control. For example, our job might take some record as the job argument, but we need to use a combination of the record identifier and record status as the foundation for the idempotency key. In these cases you can pass a `Proc` to an `acidic_by` class method:
|
200
|
+
|
201
|
+
```ruby
|
202
|
+
class ExampleJob < ActiveJob::Base
|
203
|
+
include AcidicJob
|
204
|
+
acidic_by ->(record:) { [record.id, record.status] }
|
205
|
+
|
206
|
+
def perform(record:)
|
207
|
+
# the idempotency key will be based on whatever the values of `record.id` and `record.status` are
|
208
|
+
end
|
209
|
+
end
|
210
|
+
```
|
211
|
+
|
212
|
+
> **Note:** The signature of the `acidic_by` proc _needs to match the signature_ of the job's `perform` method.
|
213
|
+
|
214
|
+
|
166
215
|
### Sidekiq Callbacks
|
167
216
|
|
168
217
|
In order to ensure that `AcidicJob::Staged` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `acidic_job` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
|
@@ -8,17 +8,19 @@ module AcidicJob
|
|
8
8
|
extend ActiveSupport::Concern
|
9
9
|
|
10
10
|
def deliver_acidicly(_options = {})
|
11
|
-
|
11
|
+
job_class = ::ActionMailer::MailDeliveryJob
|
12
12
|
|
13
13
|
job_args = [@mailer_class.name, @action.to_s, "deliver_now", @params, *@args]
|
14
14
|
# for Sidekiq, this depends on the Sidekiq::Serialization extension
|
15
|
-
serialized_job =
|
15
|
+
serialized_job = job_class.new(job_args).serialize
|
16
|
+
acidic_identifier = job_class.respond_to?(:acidic_identifier) ? job_class.acidic_identifier : :job_id
|
17
|
+
key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
|
16
18
|
|
17
19
|
AcidicJob::Run.create!(
|
18
20
|
staged: true,
|
19
|
-
job_class:
|
21
|
+
job_class: job_class.name,
|
20
22
|
serialized_job: serialized_job,
|
21
|
-
idempotency_key:
|
23
|
+
idempotency_key: key
|
22
24
|
)
|
23
25
|
end
|
24
26
|
alias deliver_transactionally deliver_acidicly
|
@@ -24,12 +24,13 @@ module AcidicJob
|
|
24
24
|
raise UnsupportedExtension unless defined?(::ActiveJob) && self < ::ActiveJob::Base
|
25
25
|
|
26
26
|
serialized_job = serialize_with_arguments(*args, **kwargs)
|
27
|
+
key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
|
27
28
|
|
28
29
|
AcidicJob::Run.create!(
|
29
30
|
staged: true,
|
30
31
|
job_class: name,
|
31
32
|
serialized_job: serialized_job,
|
32
|
-
idempotency_key:
|
33
|
+
idempotency_key: key
|
33
34
|
)
|
34
35
|
end
|
35
36
|
alias_method :perform_transactionally, :perform_acidicly
|
@@ -36,12 +36,14 @@ module AcidicJob
|
|
36
36
|
record: record
|
37
37
|
}
|
38
38
|
serialized_job = job_class.send(:job_or_instantiate, args).serialize
|
39
|
+
acidic_identifier = job_class.respond_to?(:acidic_identifier) ? job_class.acidic_identifier : :job_id
|
40
|
+
key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
|
39
41
|
|
40
42
|
AcidicJob::Run.create!(
|
41
43
|
staged: true,
|
42
44
|
job_class: job_class.name,
|
43
45
|
serialized_job: serialized_job,
|
44
|
-
idempotency_key:
|
46
|
+
idempotency_key: key
|
45
47
|
)
|
46
48
|
end
|
47
49
|
end
|
@@ -34,9 +34,10 @@ module AcidicJob
|
|
34
34
|
end
|
35
35
|
end
|
36
36
|
|
37
|
-
def serialize_job(args
|
37
|
+
def serialize_job(*args, **kwargs)
|
38
38
|
# `@args` is only set via `deserialize`; it is not a standard Sidekiq thing
|
39
39
|
arguments = args || @args
|
40
|
+
arguments += [kwargs] unless kwargs.empty?
|
40
41
|
normalized_args = ::Sidekiq.load_json(::Sidekiq.dump_json(arguments))
|
41
42
|
item = { "class" => self.class, "args" => normalized_args, "jid" => jid }
|
42
43
|
sidekiq_options = sidekiq_options_hash || {}
|
@@ -58,12 +59,13 @@ module AcidicJob
|
|
58
59
|
class_methods do
|
59
60
|
def perform_acidicly(*args, **kwargs)
|
60
61
|
serialized_job = serialize_with_arguments(*args, **kwargs)
|
62
|
+
key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
|
61
63
|
|
62
64
|
AcidicJob::Run.create!(
|
63
65
|
staged: true,
|
64
66
|
job_class: name,
|
65
67
|
serialized_job: serialized_job,
|
66
|
-
idempotency_key:
|
68
|
+
idempotency_key: key
|
67
69
|
)
|
68
70
|
end
|
69
71
|
alias_method :perform_transactionally, :perform_acidicly
|
@@ -2,15 +2,58 @@
|
|
2
2
|
|
3
3
|
module AcidicJob
|
4
4
|
class IdempotencyKey
|
5
|
-
def
|
6
|
-
|
7
|
-
|
5
|
+
def initialize(identifier = :job_id)
|
6
|
+
@identifier = identifier
|
7
|
+
end
|
8
|
+
|
9
|
+
def value_for(hash_or_job, *args, **kwargs)
|
10
|
+
value = case @identifier
|
11
|
+
when Proc
|
12
|
+
value_from_proc(hash_or_job, *args, **kwargs)
|
13
|
+
when :job_args
|
14
|
+
value_from_job_args(hash_or_job, *args, **kwargs)
|
15
|
+
else
|
16
|
+
if hash_or_job.is_a?(Hash)
|
17
|
+
value_from_job_id_for_hash(hash_or_job)
|
18
|
+
else
|
19
|
+
value_from_job_id_for_obj(hash_or_job)
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
value || value_from_job_args(hash_or_job, *args, **kwargs)
|
24
|
+
end
|
25
|
+
|
26
|
+
private
|
27
|
+
|
28
|
+
def value_from_job_id_for_hash(hash)
|
29
|
+
if hash.key?("job_id")
|
30
|
+
return if hash["job_id"].nil?
|
31
|
+
return if hash["job_id"].empty?
|
32
|
+
|
33
|
+
hash["job_id"]
|
34
|
+
elsif hash.key?("jid")
|
35
|
+
return if hash["jid"].nil?
|
36
|
+
return if hash["jid"].empty?
|
37
|
+
|
38
|
+
hash["jid"]
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def value_from_job_id_for_obj(obj)
|
43
|
+
if obj.respond_to?(:job_id)
|
44
|
+
return if obj.job_id.nil?
|
45
|
+
return if obj.job_id.empty?
|
46
|
+
|
47
|
+
obj.job_id
|
48
|
+
elsif obj.respond_to?(:jid)
|
49
|
+
return if obj.jid.nil?
|
50
|
+
return if obj.jid.empty?
|
8
51
|
|
9
|
-
|
10
|
-
return hash_or_job["job_id"]
|
52
|
+
obj.jid
|
11
53
|
end
|
12
|
-
|
54
|
+
end
|
13
55
|
|
56
|
+
def value_from_job_args(hash_or_job, *args, **kwargs)
|
14
57
|
worker_class = case hash_or_job
|
15
58
|
when Hash
|
16
59
|
hash_or_job["worker"] || hash_or_job["job_class"]
|
@@ -20,5 +63,12 @@ module AcidicJob
|
|
20
63
|
|
21
64
|
Digest::SHA1.hexdigest [worker_class, args, kwargs].flatten.join
|
22
65
|
end
|
66
|
+
|
67
|
+
def value_from_proc(_hash_or_job, *args, **kwargs)
|
68
|
+
return if args.empty? && kwargs.empty?
|
69
|
+
|
70
|
+
idempotency_args = Array(@identifier.call(*args, **kwargs))
|
71
|
+
Digest::SHA1.hexdigest idempotency_args.flatten.join
|
72
|
+
end
|
23
73
|
end
|
24
74
|
end
|
data/lib/acidic_job/staging.rb
CHANGED
@@ -26,5 +26,15 @@ module AcidicJob
|
|
26
26
|
|
27
27
|
GlobalID::Locator.locate(staged_job_gid)
|
28
28
|
end
|
29
|
+
|
30
|
+
def identifier
|
31
|
+
return jid if defined?(jid) && !jid.nil?
|
32
|
+
return job_id if defined?(job_id) && !job_id.nil?
|
33
|
+
|
34
|
+
# might be defined already in `with_acidity` method
|
35
|
+
@__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
|
36
|
+
|
37
|
+
@__acidic_job_idempotency_key
|
38
|
+
end
|
29
39
|
end
|
30
40
|
end
|
data/lib/acidic_job/version.rb
CHANGED
data/lib/acidic_job.rb
CHANGED
@@ -46,6 +46,11 @@ module AcidicJob
|
|
46
46
|
end
|
47
47
|
|
48
48
|
klass.set_callback :perform, :after, :delete_staged_job_record, if: :was_staged_job?
|
49
|
+
|
50
|
+
klass.instance_variable_set(:@acidic_identifier, :job_id)
|
51
|
+
klass.define_singleton_method(:acidic_by_job_id) { @acidic_identifier = :job_id }
|
52
|
+
klass.define_singleton_method(:acidic_by_job_args) { @acidic_identifier = :job_args }
|
53
|
+
klass.define_singleton_method(:acidic_by) { |proc| @acidic_identifier = proc }
|
49
54
|
end
|
50
55
|
|
51
56
|
included do
|
@@ -57,33 +62,47 @@ module AcidicJob
|
|
57
62
|
AcidicJob.wire_everything_up(subclass)
|
58
63
|
super
|
59
64
|
end
|
65
|
+
|
66
|
+
def acidic_identifier
|
67
|
+
@acidic_identifier
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
71
|
+
def initialize(*args, **kwargs)
|
72
|
+
# ensure this instance variable is always defined
|
73
|
+
@__acidic_job_steps = []
|
74
|
+
@__acidic_job_args = args
|
75
|
+
@__acidic_job_kwargs = kwargs
|
76
|
+
|
77
|
+
if method(__method__).super_method.arity.zero?
|
78
|
+
super()
|
79
|
+
else
|
80
|
+
super(*args, **kwargs)
|
81
|
+
end
|
60
82
|
end
|
61
83
|
|
62
84
|
def with_acidity(providing: {})
|
63
|
-
#
|
85
|
+
# ensure this instance variable is always defined
|
64
86
|
@__acidic_job_steps = []
|
65
|
-
steps
|
87
|
+
# execute the block to gather the info on what steps are defined for this job workflow
|
88
|
+
yield
|
66
89
|
|
67
90
|
# check that the block actually defined at least one step
|
68
91
|
# TODO: WRITE TESTS FOR FAULTY BLOCK VALUES
|
69
92
|
raise NoDefinedSteps if @__acidic_job_steps.nil? || @__acidic_job_steps.empty?
|
70
93
|
|
71
94
|
# convert the array of steps into a hash of recovery_points and next steps
|
72
|
-
workflow = define_workflow(
|
95
|
+
workflow = define_workflow(@__acidic_job_steps)
|
73
96
|
|
74
|
-
|
75
|
-
# might be defined already in `identifier` method
|
76
|
-
# TODO: allow idempotency to be defined by args OR job id
|
77
|
-
@__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
|
78
|
-
|
79
|
-
@run = ensure_run_record(@__acidic_job_idempotency_key, workflow, providing)
|
97
|
+
@run = ensure_run_record(workflow, providing)
|
80
98
|
|
81
99
|
# begin the workflow
|
82
100
|
process_run(@run)
|
83
101
|
end
|
84
102
|
|
85
103
|
# DEPRECATED
|
86
|
-
def idempotently(with
|
104
|
+
def idempotently(with: {}, &blk)
|
105
|
+
ActiveSupport::Deprecation.new("1.0", "AcidicJob").deprecation_warning(:idempotently)
|
87
106
|
with_acidity(providing: with, &blk)
|
88
107
|
end
|
89
108
|
|
@@ -93,11 +112,16 @@ module AcidicJob
|
|
93
112
|
FinishedPoint.new
|
94
113
|
end
|
95
114
|
|
115
|
+
# TODO: allow idempotency to be defined by args OR job id
|
96
116
|
# rubocop:disable Naming/MemoizedInstanceVariableName
|
97
117
|
def idempotency_key
|
98
|
-
|
118
|
+
if defined?(@__acidic_job_idempotency_key) && !@__acidic_job_idempotency_key.nil?
|
119
|
+
return @__acidic_job_idempotency_key
|
120
|
+
end
|
99
121
|
|
100
|
-
|
122
|
+
acidic_identifier = self.class.acidic_identifier
|
123
|
+
@__acidic_job_idempotency_key ||= IdempotencyKey.new(acidic_identifier)
|
124
|
+
.value_for(self, *@__acidic_job_args, **@__acidic_job_kwargs)
|
101
125
|
end
|
102
126
|
# rubocop:enable Naming/MemoizedInstanceVariableName
|
103
127
|
|
@@ -171,7 +195,7 @@ module AcidicJob
|
|
171
195
|
# { "step 1": { does: "step 1", awaits: [], then: "step 2" }, ... }
|
172
196
|
end
|
173
197
|
|
174
|
-
def ensure_run_record(
|
198
|
+
def ensure_run_record(workflow, accessors)
|
175
199
|
isolation_level = case ActiveRecord::Base.connection.adapter_name.downcase.to_sym
|
176
200
|
when :sqlite
|
177
201
|
:read_uncommitted
|
@@ -180,8 +204,8 @@ module AcidicJob
|
|
180
204
|
end
|
181
205
|
|
182
206
|
ActiveRecord::Base.transaction(isolation: isolation_level) do
|
183
|
-
run = Run.find_by(idempotency_key:
|
184
|
-
serialized_job = serialize_job(
|
207
|
+
run = Run.find_by(idempotency_key: idempotency_key)
|
208
|
+
serialized_job = serialize_job(*@__acidic_job_args, **@__acidic_job_kwargs)
|
185
209
|
|
186
210
|
if run.present?
|
187
211
|
# Programs enqueuing multiple jobs with different parameters but the
|
@@ -201,7 +225,7 @@ module AcidicJob
|
|
201
225
|
else
|
202
226
|
run = Run.create!(
|
203
227
|
staged: false,
|
204
|
-
idempotency_key:
|
228
|
+
idempotency_key: idempotency_key,
|
205
229
|
job_class: self.class.name,
|
206
230
|
locked_at: Time.current,
|
207
231
|
last_run_at: Time.current,
|
@@ -246,14 +270,4 @@ module AcidicJob
|
|
246
270
|
|
247
271
|
true
|
248
272
|
end
|
249
|
-
|
250
|
-
def identifier
|
251
|
-
return jid if defined?(jid) && !jid.nil?
|
252
|
-
return job_id if defined?(job_id) && !job_id.nil?
|
253
|
-
|
254
|
-
# might be defined already in `with_acidity` method
|
255
|
-
@__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
|
256
|
-
|
257
|
-
@__acidic_job_idempotency_key
|
258
|
-
end
|
259
273
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: acidic_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.0.
|
4
|
+
version: 1.0.0.pre12
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- fractaledmind
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-03-
|
11
|
+
date: 2022-03-14 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|
@@ -63,6 +63,7 @@ files:
|
|
63
63
|
- ".gitignore"
|
64
64
|
- ".rubocop.yml"
|
65
65
|
- ".ruby_version"
|
66
|
+
- ".tool-versions"
|
66
67
|
- Gemfile
|
67
68
|
- Gemfile.lock
|
68
69
|
- LICENSE
|