acidic_job 1.0.0.pre9 → 1.0.0.pre12

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3971aa5af368952620c483730c9a8a28efd9e01bffecc38d015d322e165bdffe
-  data.tar.gz: a6c7374d7e0f2121a0fc92b988c28cee29d488f32506dd84e9e84c15b0117eda
+  metadata.gz: 1af7bca23aa5ead8fed158fb73565736555236bf4515ab73f631a3013e138f12
+  data.tar.gz: cc1118240efe70d71dcd7c2eb07435f3e56934c2c3e8653848a9c3467dc60b83
 SHA512:
-  metadata.gz: 7a890818e15da5fb4f5990f9753545d2e33c2bfb6d6e2a9793a42efeed34fcb6dbef076b18111980b0bfe32fe58f490f5fad0d70640b8d6624c8ae0210b4dd8a
-  data.tar.gz: b988408714e535184d553982a8220e32c83e7551ea129a7a5c001f1e83462d03d64f86e7df4112fed3af51f1a8a197f38c587fd01a9dc0b4d77ea8051a43aee0
+  metadata.gz: 90073b7d2964d65886ca2fd4320481455e04218db92ab3b2ccd65bdffc527b03f8c928c58590ed62cf7734c1702687978dd5f1406ea740f13f1fbccb7c27b48d
+  data.tar.gz: 9f2910c1620f489a1bb5a310821ead5363671aa0ffc9a21f909894f43c2de52b056ee096f17455fb8b34bd55d8f521ca71760df3959b8b0c1f90562c50129ff4

data/.tool-versions

@@ -0,0 +1 @@
+ruby 3.0.2

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    acidic_job (1.0.0.pre9)
+    acidic_job (1.0.0.pre12)
       activerecord (>= 6.1.0)
       activesupport
 

data/README.md

@@ -1,5 +1,8 @@
 # AcidicJob
 
+[![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://badge.fury.io/rb/acidic_job)
+![main workflow](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
+
 ### Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
 
 At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
@@ -64,6 +67,7 @@ It provides a suite of functionality that empowers you to create complex, robust
 * Transactional Steps — break your job into a series of steps, each of which will be run within an acidic database transaction, allowing retries to jump back to the last "recovery point".
 * Persisted Attributes — when retrying jobs at later steps, we need to ensure that data created in previous steps is still available to later steps on retry.
 * Transactionally Staged Jobs — enqueue additional jobs within the acidic transaction safely
+* Custom Idempotency Keys — use something other than the job ID for the idempotency key of the job run
 * Sidekiq Callbacks — bring ActiveJob-like callbacks into your pure Sidekiq Workers
 * Sidekiq Batches — leverage the power of Sidekiq Pro's `batch` functionality without the hassle
 
@@ -163,6 +167,51 @@ class RideCreateJob < ActiveJob::Base
 end
 ```
 
+### Custom Idempotency Keys
+
+By default, `AcidicJob` uses the job identifier provided by the queueing system (ActiveJob or Sidekiq) as the idempotency key for the job run. The idempotency key is what is used to guarantee that no two runs of the same job occur. However, sometimes we need particular jobs to be idempotent based on some other criteria. In these cases, `AcidicJob` provides a collection of tools to allow you to ensure the idempotency of your jobs.
+
+Firstly, you can configure your job class to explicitly use either the job identifier or the job arguments as the foundation for the idempotency key. A job class that calls the `acidic_by_job_id` class method (which is the default behavior) will simply make the job run's idempotency key the job's identifier:
+
+```ruby
+class ExampleJob < ActiveJob::Base
+  include AcidicJob
+  acidic_by_job_id
+
+  def perform
+  end
+end
+```
+
+Conversely, a job class can use the `acidic_by_job_args` method to configure that job class to use the arguments passed to the job as the foundation for the job run's idempotency key:
+
+```ruby
+class ExampleJob < ActiveJob::Base
+  include AcidicJob
+  acidic_by_job_args
+
+  def perform(arg_1, arg_2)
+    # the idempotency key will be based on whatever the values of `arg_1` and `arg_2` are
+  end
+end
+```
+
+These options cover the two common situations, but sometimes our systems need finer-grained control. For example, our job might take some record as the job argument, but we need to use a combination of the record identifier and record status as the foundation for the idempotency key. In these cases you can pass a `Proc` to an `acidic_by` class method:
+
+```ruby
+class ExampleJob < ActiveJob::Base
+  include AcidicJob
+  acidic_by ->(record:) { [record.id, record.status] }
+
+  def perform(record:)
+    # the idempotency key will be based on whatever the values of `record.id` and `record.status` are
+  end
+end
+```
+
+> **Note:** The signature of the `acidic_by` proc _needs to match the signature_ of the job's `perform` method.
+
+
 ### Sidekiq Callbacks
 
 In order to ensure that `AcidicJob::Staged` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `acidic_job` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).

data/lib/acidic_job/extensions/action_mailer.rb

@@ -8,17 +8,19 @@ module AcidicJob
       extend ActiveSupport::Concern
 
       def deliver_acidicly(_options = {})
-        job = ::ActionMailer::MailDeliveryJob
+        job_class = ::ActionMailer::MailDeliveryJob
 
         job_args = [@mailer_class.name, @action.to_s, "deliver_now", @params, *@args]
         # for Sidekiq, this depends on the Sidekiq::Serialization extension
-        serialized_job = job.new(job_args).serialize
+        serialized_job = job_class.new(job_args).serialize
+        acidic_identifier = job_class.respond_to?(:acidic_identifier) ? job_class.acidic_identifier : :job_id
+        key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
 
         AcidicJob::Run.create!(
           staged: true,
-          job_class: job.name,
+          job_class: job_class.name,
           serialized_job: serialized_job,
-          idempotency_key: IdempotencyKey.value_for(serialized_job)
+          idempotency_key: key
         )
       end
       alias deliver_transactionally deliver_acidicly

data/lib/acidic_job/extensions/active_job.rb

@@ -24,12 +24,13 @@ module AcidicJob
           raise UnsupportedExtension unless defined?(::ActiveJob) && self < ::ActiveJob::Base
 
           serialized_job = serialize_with_arguments(*args, **kwargs)
+          key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
 
           AcidicJob::Run.create!(
             staged: true,
             job_class: name,
             serialized_job: serialized_job,
-            idempotency_key: IdempotencyKey.value_for(serialized_job)
+            idempotency_key: key
           )
         end
         alias_method :perform_transactionally, :perform_acidicly

data/lib/acidic_job/extensions/noticed.rb

@@ -36,12 +36,14 @@ module AcidicJob
               record: record
             }
             serialized_job = job_class.send(:job_or_instantiate, args).serialize
+            acidic_identifier = job_class.respond_to?(:acidic_identifier) ? job_class.acidic_identifier : :job_id
+            key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
 
             AcidicJob::Run.create!(
               staged: true,
               job_class: job_class.name,
               serialized_job: serialized_job,
-              idempotency_key: IdempotencyKey.value_for(serialized_job)
+              idempotency_key: key
             )
           end
         end

data/lib/acidic_job/extensions/sidekiq.rb

@@ -34,9 +34,10 @@ module AcidicJob
           end
         end
 
-        def serialize_job(args = [], _kwargs = nil)
+        def serialize_job(*args, **kwargs)
           # `@args` is only set via `deserialize`; it is not a standard Sidekiq thing
           arguments = args || @args
+          arguments += [kwargs] unless kwargs.empty?
           normalized_args = ::Sidekiq.load_json(::Sidekiq.dump_json(arguments))
           item = { "class" => self.class, "args" => normalized_args, "jid" => jid }
           sidekiq_options = sidekiq_options_hash || {}
@@ -58,12 +59,13 @@ module AcidicJob
         class_methods do
           def perform_acidicly(*args, **kwargs)
             serialized_job = serialize_with_arguments(*args, **kwargs)
+            key = IdempotencyKey.new(acidic_identifier).value_for(serialized_job)
 
             AcidicJob::Run.create!(
               staged: true,
               job_class: name,
               serialized_job: serialized_job,
-              idempotency_key: IdempotencyKey.value_for(serialized_job)
+              idempotency_key: key
             )
           end
           alias_method :perform_transactionally, :perform_acidicly

data/lib/acidic_job/idempotency_key.rb

@@ -2,15 +2,58 @@
 
 module AcidicJob
   class IdempotencyKey
-    def self.value_for(hash_or_job, *args, **kwargs)
-      return hash_or_job.job_id if hash_or_job.respond_to?(:job_id) && !hash_or_job.job_id.nil?
-      return hash_or_job.jid if hash_or_job.respond_to?(:jid) && !hash_or_job.jid.nil?
+    def initialize(identifier = :job_id)
+      @identifier = identifier
+    end
+
+    def value_for(hash_or_job, *args, **kwargs)
+      value = case @identifier
+              when Proc
+                value_from_proc(hash_or_job, *args, **kwargs)
+              when :job_args
+                value_from_job_args(hash_or_job, *args, **kwargs)
+              else
+                if hash_or_job.is_a?(Hash)
+                  value_from_job_id_for_hash(hash_or_job)
+                else
+                  value_from_job_id_for_obj(hash_or_job)
+                end
+              end
+
+      value || value_from_job_args(hash_or_job, *args, **kwargs)
+    end
+
+    private
+
+    def value_from_job_id_for_hash(hash)
+      if hash.key?("job_id")
+        return if hash["job_id"].nil?
+        return if hash["job_id"].empty?
+
+        hash["job_id"]
+      elsif hash.key?("jid")
+        return if hash["jid"].nil?
+        return if hash["jid"].empty?
+
+        hash["jid"]
+      end
+    end
+
+    def value_from_job_id_for_obj(obj)
+      if obj.respond_to?(:job_id)
+        return if obj.job_id.nil?
+        return if obj.job_id.empty?
+
+        obj.job_id
+      elsif obj.respond_to?(:jid)
+        return if obj.jid.nil?
+        return if obj.jid.empty?
 
-      if hash_or_job.is_a?(Hash) && hash_or_job.key?("job_id") && !hash_or_job["job_id"].nil?
-        return hash_or_job["job_id"]
+        obj.jid
       end
-      return hash_or_job["jid"] if hash_or_job.is_a?(Hash) && hash_or_job.key?("jid") && !hash_or_job["jid"].nil?
+    end
 
+    def value_from_job_args(hash_or_job, *args, **kwargs)
       worker_class = case hash_or_job
                      when Hash
                        hash_or_job["worker"] || hash_or_job["job_class"]
@@ -20,5 +63,12 @@ module AcidicJob
 
       Digest::SHA1.hexdigest [worker_class, args, kwargs].flatten.join
     end
+
+    def value_from_proc(_hash_or_job, *args, **kwargs)
+      return if args.empty? && kwargs.empty?
+
+      idempotency_args = Array(@identifier.call(*args, **kwargs))
+      Digest::SHA1.hexdigest idempotency_args.flatten.join
+    end
   end
 end

data/lib/acidic_job/staging.rb

@@ -26,5 +26,15 @@ module AcidicJob
 
       GlobalID::Locator.locate(staged_job_gid)
     end
+
+    def identifier
+      return jid if defined?(jid) && !jid.nil?
+      return job_id if defined?(job_id) && !job_id.nil?
+
+      # might be defined already in `with_acidity` method
+      @__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
+
+      @__acidic_job_idempotency_key
+    end
   end
 end

data/lib/acidic_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module AcidicJob
-  VERSION = "1.0.0.pre9"
+  VERSION = "1.0.0.pre12"
 end

data/lib/acidic_job.rb

@@ -46,6 +46,11 @@ module AcidicJob
     end
 
     klass.set_callback :perform, :after, :delete_staged_job_record, if: :was_staged_job?
+
+    klass.instance_variable_set(:@acidic_identifier, :job_id)
+    klass.define_singleton_method(:acidic_by_job_id) { @acidic_identifier = :job_id }
+    klass.define_singleton_method(:acidic_by_job_args) { @acidic_identifier = :job_args }
+    klass.define_singleton_method(:acidic_by) { |proc| @acidic_identifier = proc }
   end
 
   included do
@@ -57,33 +62,47 @@ module AcidicJob
       AcidicJob.wire_everything_up(subclass)
       super
     end
+
+    def acidic_identifier
+      @acidic_identifier
+    end
+  end
+
+  def initialize(*args, **kwargs)
+    # ensure this instance variable is always defined
+    @__acidic_job_steps = []
+    @__acidic_job_args = args
+    @__acidic_job_kwargs = kwargs
+
+    if method(__method__).super_method.arity.zero?
+      super()
+    else
+      super(*args, **kwargs)
+    end
   end
 
   def with_acidity(providing: {})
-    # execute the block to gather the info on what steps are defined for this job workflow
+    # ensure this instance variable is always defined
     @__acidic_job_steps = []
-    steps = yield || []
+    # execute the block to gather the info on what steps are defined for this job workflow
+    yield
 
     # check that the block actually defined at least one step
     # TODO: WRITE TESTS FOR FAULTY BLOCK VALUES
     raise NoDefinedSteps if @__acidic_job_steps.nil? || @__acidic_job_steps.empty?
 
     # convert the array of steps into a hash of recovery_points and next steps
-    workflow = define_workflow(steps)
+    workflow = define_workflow(@__acidic_job_steps)
 
-    # determine the idempotency key value for this job run (`job_id` or `jid`)
-    # might be defined already in `identifier` method
-    # TODO: allow idempotency to be defined by args OR job id
-    @__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
-
-    @run = ensure_run_record(@__acidic_job_idempotency_key, workflow, providing)
+    @run = ensure_run_record(workflow, providing)
 
     # begin the workflow
     process_run(@run)
   end
 
   # DEPRECATED
-  def idempotently(with:, &blk)
+  def idempotently(with: {}, &blk)
+    ActiveSupport::Deprecation.new("1.0", "AcidicJob").deprecation_warning(:idempotently)
     with_acidity(providing: with, &blk)
   end
 
@@ -93,11 +112,16 @@ module AcidicJob
     FinishedPoint.new
   end
 
+  # TODO: allow idempotency to be defined by args OR job id
   # rubocop:disable Naming/MemoizedInstanceVariableName
   def idempotency_key
-    return @__acidic_job_idempotency_key if defined? @__acidic_job_idempotency_key
+    if defined?(@__acidic_job_idempotency_key) && !@__acidic_job_idempotency_key.nil?
+      return @__acidic_job_idempotency_key
+    end
 
-    @__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
+    acidic_identifier = self.class.acidic_identifier
+    @__acidic_job_idempotency_key ||= IdempotencyKey.new(acidic_identifier)
+                                                    .value_for(self, *@__acidic_job_args, **@__acidic_job_kwargs)
   end
   # rubocop:enable Naming/MemoizedInstanceVariableName
 
@@ -171,7 +195,7 @@ module AcidicJob
     # { "step 1": { does: "step 1", awaits: [], then: "step 2" }, ...  }
   end
 
-  def ensure_run_record(key_val, workflow, accessors)
+  def ensure_run_record(workflow, accessors)
     isolation_level = case ActiveRecord::Base.connection.adapter_name.downcase.to_sym
                       when :sqlite
                         :read_uncommitted
@@ -180,8 +204,8 @@ module AcidicJob
                       end
 
     ActiveRecord::Base.transaction(isolation: isolation_level) do
-      run = Run.find_by(idempotency_key: key_val)
-      serialized_job = serialize_job(@__acidic_job_args, @__acidic_job_kwargs)
+      run = Run.find_by(idempotency_key: idempotency_key)
+      serialized_job = serialize_job(*@__acidic_job_args, **@__acidic_job_kwargs)
 
       if run.present?
         # Programs enqueuing multiple jobs with different parameters but the
@@ -201,7 +225,7 @@ module AcidicJob
       else
         run = Run.create!(
           staged: false,
-          idempotency_key: key_val,
+          idempotency_key: idempotency_key,
           job_class: self.class.name,
           locked_at: Time.current,
           last_run_at: Time.current,
@@ -246,14 +270,4 @@ module AcidicJob
 
     true
   end
-
-  def identifier
-    return jid if defined?(jid) && !jid.nil?
-    return job_id if defined?(job_id) && !job_id.nil?
-
-    # might be defined already in `with_acidity` method
-    @__acidic_job_idempotency_key ||= IdempotencyKey.value_for(self, @__acidic_job_args, @__acidic_job_kwargs)
-
-    @__acidic_job_idempotency_key
-  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: acidic_job
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre9
+  version: 1.0.0.pre12
 platform: ruby
 authors:
 - fractaledmind
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-03-12 00:00:00.000000000 Z
+date: 2022-03-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -63,6 +63,7 @@ files:
 - ".gitignore"
 - ".rubocop.yml"
 - ".ruby_version"
+- ".tool-versions"
 - Gemfile
 - Gemfile.lock
 - LICENSE