acidic_job 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 739d9a393b9dfcf91c6e941e188cab965a2800cf7a729e70a5d30542c76b1237
-  data.tar.gz: 6fb81e32e31e6db7001a11d607f33d3bc7f1cb5ddc147d48f8ead84ae15fc7d1
+  metadata.gz: f6f61f9992e6528f4e7c1b6a5b774cd1fa1557b4f069fbd14b5e415298dbde15
+  data.tar.gz: 46e279aca2bdc5e2379b496374334f0687dff92c845c2c4725d7e7fa7b96b571
 SHA512:
-  metadata.gz: bc2d1796950ffb4640d49dc61e05771b2a72f6b6776d96dda4fa35f93720e0687102e08bc8ad39a176d44be828528048df03c4492cbd0b1d121865ef929e5b8f
-  data.tar.gz: 32db6d94ca8a8a70d48a7afa863f85f50ebdf05320c9461b1f8c647d4681e697d7d5bc814864fb8846b516482db7057e8659598387f661e43764115f1525e840
+  metadata.gz: 8042feab92bccc37ad4006145c7c1c780da1eb94cf24f11dfc6808ce95edc18c986ca2424c4aa9c468444267da21ff103089ad0a7e9577a7e6f3e32d097daba7
+  data.tar.gz: fc7ea7a48fc6560c3984538d9c8d2dd681492f22bd60bd4104d4db39a1bed53b950a386384f4db43caf22bc72ab0c3c9b39c7e36eb4232f646ee6e1ef7885430

data/.gitignore

@@ -8,3 +8,4 @@
 /tmp/
 .DS_Store
 /test/database.sqlite
+slides.md

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    acidic_job (0.6.0)
+    acidic_job (0.7.0)
       activerecord (>= 4.0.0)
       activesupport
 

data/README.md

@@ -1,8 +1,8 @@
 # AcidicJob
 
-### Idempotent operations for Rails apps, built on top of ActiveJob.
+### Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
 
-At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`. With `ActiveJob`, we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
+At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
 
 However, in order to ensure that our operational jobs are _robust_, we need to ensure that they are properly [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). As stated in the [GitLab Sidekiq Style Guide](https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#idempotent-jobs):
 
@@ -37,7 +37,7 @@ Or simply execute to install the gem yourself:
 
     $ bundle add acidic_job
 
-Then, use the following command to copy over the AcidicJobKey migration.
+Then, use the following command to copy over the `AcidicJob::Key` migration file as well as the `AcidicJob::Staged` migration file.
 
 ```
 rails generate acidic_job
@@ -45,7 +45,19 @@ rails generate acidic_job
 
 ## Usage
 
-`AcidicJob` is a concern that you `include` into your operation jobs which provides two public methods to help you make your jobs idempotent and robust—`idempotently` and `step`. You can see them "in action" in the example job below:
+`AcidicJob` is a concern that you `include` into your operation jobs.
+
+```ruby
+class RideCreateJob < ActiveJob::Base
+  include AcidicJob
+end
+```
+
+It provides a suite of functionality that empowers you to create complex, robust, and _acidic_ jobs.
+
+### Transactional Steps
+
+The first and foundational feature `acidic_job` provides is the `idempotently` method, which takes a block of transactional step methods (defined via the `step`) method:
 
 ```ruby
 class RideCreateJob < ActiveJob::Base
@@ -75,7 +87,75 @@ end
 
 `idempotently` takes only the `with:` named parameter and a block where you define the steps of this operation. `step` simply takes the name of a method available in the job. That's all!
 
-So, how does `AcidicJob` make this operation idempotent and robust then? In simplest form, `AcidicJob` creates an "idempotency key" record for each job run, where it stores information about that job run, like the parameters passed in and the step the job is on. It then wraps each of your step methods in a database transaction to ensure that each step in the operation is transactionally secure. Finally, it handles a variety of edge-cases and error conditions for you as well. But, basically, by explicitly breaking your operation into steps and storing a record of each job run and updating its current step as it runs, we level up the `ActiveJob` retry mechanism to ensure that we don't retry already finished steps if something goes wrong and the job has to retry. Then, by wrapping each step in a transaction, we ensure each individual step is ACIDic. Taken together, these two strategies help us to ensure that our operational jobs are both idempotent and ACIDic.
+Now, each execution of this job will find or create an `AcidicJob::Key` record, which we leverage to wrap every step in a database transaction. Moreover, this database record allows `acidic_job` to ensure that if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, and won't even execute the first two step methods. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
+
+### Persisted Attributes
+
+Any objects passed to the `with` option on the `idempotently` method are not just made available to each of your step methods, they are made available across retries. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is done by serializing all objects to a field on the `AcidicJob::Key` and manually providing getters and setters that sync with the database record.
+
+```ruby
+class RideCreateJob < ActiveJob::Base
+  include AcidicJob
+
+  def perform(ride_params)
+    idempotently with: { ride: nil } do
+      step :create_ride_and_audit_record
+      step :create_stripe_charge
+      step :send_receipt
+    end
+  end
+
+  def create_ride_and_audit_record
+    self.ride = Ride.create!
+  end
+
+  def create_stripe_charge
+    Stripe::Charge.create(amount: 20_00, customer: @ride.user)
+  end
+
+  # ...
+end
+```
+
+**Note:** This does mean that you are restricted to objects that can be serialized by ActiveRecord, thus no Procs, for example.
+
+**Note:** You will note the use of `self.ride = ...` in the code sample above. In order to call the attribute setter method that will sync with the database record, you _must_ use this style. `@ride = ...` and/or `ride = ...` will both fail to sync the value with the datbase record.
+
+### Transactionally Staged Jobs
+
+A standard problem when inside of database transactions is enqueuing other jobs. On the one hand, you could enqueue a job inside of a transaction that then rollbacks, which would leave that job to fail and retry and fail. On the other hand, you could enqueue a job that is picked up before the transaction commits, which would mean the records are not yet available to this job.
+
+In order to mitigate against such issues without forcing you to use a database-backed job queue, `acidic_job` provides `perform_transactionally` and `deliver_transactionally` methods to "transactionally stage" enqueuing other jobs from within a step (whether another ActiveJob or a Sidekiq::Worker or an ActionMailer delivery). These methods will create a new `AcidicJob::Staged` record, but inside of the database transaction of the `step`. Upon commit of that transaction, a model callback pushes the job to your actual job queue.  Once the job has been successfully performed, the `AcidicJob::Staged` record is deleted so that this table doesn't grow unbounded and unnecessarily.
+
+```ruby
+class RideCreateJob < ActiveJob::Base
+  include AcidicJob
+
+  def perform(ride_params)
+    idempotently with: { user: current_user, params: ride_params, ride: nil } do
+      step :create_ride_and_audit_record
+      step :create_stripe_charge
+      step :send_receipt
+    end
+  end
+
+  # ...
+
+  def send_receipt
+    RideMailer.with(ride: @ride, user: @user).confirm_charge.delivery_transactionally
+  end
+end
+```
+
+### Sidekiq Callbacks
+
+In order to ensure that `AcidicJob::Staged` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `acidic_job` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
+
+This allows `acidic_job` to use an `after_perform` callback to delete the `AcidicJob::Staged` record, whether you are using the gem with ActiveJob or pure Sidekiq Workers. Of course, this means that you can add your own callbacks to any jobs or workers that include the `AcidicJob` module as well.
+
+### Sidekiq Batches
+
+One final feature for those of you using Sidekiq Pro: an integrated DSL for Sidekiq Batches. By simply adding the `awaits` option to your step declarations, you can attach any number of additional, asynchronous workers to your step. This is profoundly powerful, as it means that you can define a workflow where step 2 is started _if and only if_ step 1 succeeds, but step 1 can have 3 different workers enqueued on 3 different queues, each running in parallel. Once all 3 workers succeed, `acidic_job` will move on to step 2. That's right, by leveraging the power of Sidekiq Batches, you can have workers that are executed in parallel, on separate queues, and asynchronously, but are still blocking—as a group—the next step in your workflow! This unlocks incredible power and flexibility for defining and structuring complex workflows and operations, and in my mind is the number one selling point for Sidekiq Pro.
 
 ## Development
 

data/lib/acidic_job/deliver_transactionally_extension.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  module DeliverTransactionallyExtension
+    def deliver_transactionally(options = {})
+      job = delivery_job_class
+
+      attributes = {
+        adapter: "activejob",
+        job_name: job.name
+      }
+
+      job_args = if job <= ActionMailer::Parameterized::MailDeliveryJob
+        [@mailer_class.name, @action.to_s, "deliver_now", {params: @params, args: @args}]
+      else
+        [@mailer_class.name, @action.to_s, "deliver_now", @params, *@args]
+      end
+
+      attributes[:job_args] = job.new(job_args).serialize
+
+      AcidicJob::Staged.create!(attributes)
+    end
+  end
+end

data/lib/acidic_job/errors.rb

@@ -14,4 +14,6 @@ module AcidicJob
   class SerializedTransactionConflict < Error; end
 
   class UnknownJobAdapter < Error; end
+
+  class NoDefinedSteps < Error; end
 end

data/lib/acidic_job/key.rb

@@ -10,6 +10,7 @@ module AcidicJob
 
     serialize :error_object
     serialize :job_args
+    serialize :workflow
     store :attr_accessors
 
     validates :idempotency_key, presence: true, uniqueness: { scope: %i[job_name job_args] }

data/lib/acidic_job/perform_wrapper.rb

@@ -3,6 +3,23 @@
 module AcidicJob
   module PerformWrapper
     def perform(*args, **kwargs)
+      # extract the `staged_job_gid` if present
+      # so that we can later delete the record in an `after_perform` callback
+      final_arg = args.last
+      if final_arg.is_a?(Hash) && final_arg.key?("staged_job_gid")
+        args = args[0..-2]
+        @staged_job_gid = final_arg["staged_job_gid"]
+      end
+
+      set_arguments_for_perform(*args, **kwargs)
+
+      super(*args, **kwargs)
+    end
+
+    private
+
+    # rubocop:disable Metrics/AbcSize
+    def set_arguments_for_perform(*args, **kwargs)
       # store arguments passed into `perform` so that we can later persist
       # them to `AcidicJob::Key#job_args` for both ActiveJob and Sidekiq::Worker
       @arguments_for_perform = if args.any? && kwargs.any?
@@ -14,8 +31,7 @@ module AcidicJob
                                else
                                  []
                                end
-
-      super
     end
+    # rubocop:enable Metrics/AbcSize
   end
 end

data/lib/acidic_job/sidekiq_callbacks.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require "active_support/callbacks"
+
+# Following approach used by ActiveJob
+# https://github.com/rails/rails/blob/93c9534c9871d4adad4bc33b5edc355672b59c61/activejob/lib/active_job/callbacks.rb
+module SidekiqCallbacks
+  extend ActiveSupport::Concern
+
+  def self.prepended(base)
+    base.include(ActiveSupport::Callbacks)
+
+    # Check to see if we already have any callbacks for :perform
+    # Prevents overwriting callbacks if we already included this module (and defined callbacks)
+    base.define_callbacks :perform unless base.respond_to?(:_perform_callbacks) && base._perform_callbacks.present?
+
+    class << base
+      prepend ClassMethods
+    end
+  end
+
+  def perform(*args)
+    if respond_to?(:run_callbacks)
+      run_callbacks :perform do
+        super(*args)
+      end
+    else
+      super(*args)
+    end
+  end
+
+  module ClassMethods
+    def around_perform(*filters, &blk)
+      set_callback(:perform, :around, *filters, &blk)
+    end
+
+    def before_perform(*filters, &blk)
+      set_callback(:perform, :before, *filters, &blk)
+    end
+
+    def after_perform(*filters, &blk)
+      set_callback(:perform, :after, *filters, &blk)
+    end
+  end
+end

data/lib/acidic_job/staged.rb

@@ -1,11 +1,14 @@
 # frozen_string_literal: true
 
 require "active_record"
+require "global_id"
 
 module AcidicJob
   class Staged < ActiveRecord::Base
     self.table_name = "staged_acidic_jobs"
 
+    include GlobalID::Identification
+
     validates :adapter, presence: true
     validates :job_name, presence: true
     validates :job_args, presence: true
@@ -14,19 +17,34 @@ module AcidicJob
 
     after_create_commit :enqueue_job
 
+    private
+
+    # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
     def enqueue_job
+      gid = { "staged_job_gid" => to_global_id.to_s }
+
+      if job_args.is_a?(Hash) && job_args.key?("arguments")
+        job_args["arguments"].concat([gid])
+      else
+        job_args.concat([gid])
+      end
+
       case adapter
       when "activejob"
         job = ActiveJob::Base.deserialize(job_args)
         job.enqueue
       when "sidekiq"
-        Sidekiq::Client.push("class" => job_name, "args" => job_args)
+        Sidekiq::Client.push(
+          "class" => job_name,
+          "args" => job_args
+        )
       else
         raise UnknownJobAdapter.new(adapter: adapter)
       end
 
-      # TODO: ensure successful enqueuing before deletion
-      delete
+      # NOTE: record will be deleted after the job has successfully been performed
+      true
     end
+    # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
   end
 end

data/lib/acidic_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module AcidicJob
-  VERSION = "0.6.0"
+  VERSION = "0.7.0"
 end

data/lib/acidic_job.rb

@@ -9,6 +9,7 @@ require_relative "acidic_job/key"
 require_relative "acidic_job/staged"
 require_relative "acidic_job/perform_wrapper"
 require_relative "acidic_job/perform_transactionally_extension"
+require_relative "acidic_job/sidekiq_callbacks"
 require "active_support/concern"
 
 # rubocop:disable Metrics/ModuleLength, Metrics/AbcSize, Metrics/MethodLength
@@ -17,13 +18,22 @@ module AcidicJob
 
   def self.wire_everything_up(klass)
     klass.attr_reader :key
-    klass.attr_accessor :arguments_for_perform
+    klass.attr_reader :staged_job_gid
+    klass.attr_reader :arguments_for_perform
 
     # Extend ActiveJob with `perform_transactionally` class method
     klass.include PerformTransactionallyExtension
 
+    if defined?(ActionMailer)
+      ActionMailer::Parameterized::MessageDelivery.include DeliverTransactionallyExtension
+    end
+
     # Ensure our `perform` method always runs first to gather parameters
     klass.prepend PerformWrapper
+
+    klass.prepend SidekiqCallbacks unless klass.respond_to?(:after_perform)
+
+    klass.after_perform :delete_staged_job_record, if: :staged_job_gid
   end
 
   included do
@@ -35,6 +45,14 @@ module AcidicJob
       AcidicJob.wire_everything_up(subclass)
       super
     end
+
+    def initiate(*args)
+      operation = Sidekiq::Batch.new
+      operation.on(:success, self, *args)
+      operation.jobs do
+        perform_async
+      end
+    end
   end
 
   # Number of seconds passed which we consider a held idempotency key lock to be
@@ -45,43 +63,54 @@ module AcidicJob
 
   # takes a block
   def idempotently(with:)
-    # execute the block to gather the info on what phases are defined for this job
-    defined_steps = yield
-    # [:create_ride_and_audit_record, :create_stripe_charge, :send_receipt]
+    # execute the block to gather the info on what steps are defined for this job workflow
+    steps = yield || []
 
-    # convert the array of steps into a hash of recovery_points and callable actions
-    phases = define_atomic_phases(defined_steps)
-    # { create_ride_and_audit_record: <#Method >, ... }
+    raise NoDefinedSteps if steps.empty?
 
-    # find or create an Key record (our idempotency key) to store all information about this job
-    # side-effect: will set the @key instance variable
+    # convert the array of steps into a hash of recovery_points and next steps
+    workflow = define_workflow(steps)
+
+    # find or create a Key record (our idempotency key) to store all information about this job
     #
     # A key concept here is that if two requests try to insert or update within
     # close proximity, one of the two will be aborted by Postgres because we're
     # using a transaction with SERIALIZABLE isolation level. It may not look
     # it, but this code is safe from races.
-    ensure_idempotency_key_record(idempotency_key_value, defined_steps.first)
+    key = ensure_idempotency_key_record(idempotency_key_value, workflow, with)
+
+    # begin the workflow
+    process_key(key)
+  end
+
+  def process_key(key)
+    @key = key
 
     # if the key record is already marked as finished, immediately return its result
     return @key.succeeded? if @key.finished?
 
-    # set accessors for each argument passed in to ensure they are available
-    # to the step methods the job will have written
-    # THIS HAPPENS OUTSIDE OF ANY TRANSACTION
-    define_accessors_for_passed_arguments(with, @key)
-
-    # otherwise, we will enter a loop to process each required step of the job
-    phases.size.times do
-      # our `phases` hash uses Symbols for keys
-      recovery_point = @key.recovery_point.to_sym
+    # otherwise, we will enter a loop to process each step of the workflow
+    @key.workflow.size.times do
+      recovery_point = @key.recovery_point.to_s
+      current_step = @key.workflow[recovery_point]
 
-      case recovery_point
-      when Key::RECOVERY_POINT_FINISHED.to_sym
+      if recovery_point == Key::RECOVERY_POINT_FINISHED.to_s
         break
+      elsif current_step.nil?
+        raise UnknownRecoveryPoint, "Defined workflow does not reference this step: #{recovery_point}"
+      elsif (jobs = current_step.fetch("awaits", [])).any?
+        acidic_step @key, current_step
+        # THIS MUST BE DONE AFTER THE KEY RECOVERY POINT HAS BEEN UPDATED
+        enqueue_step_parallel_jobs(jobs)
+        # after processing the current step, break the processing loop
+        # and stop this method from blocking in the primary worker
+        # as it will continue once the background workers all succeed
+        # so we want to keep the primary worker queue free to process new work
+        # this CANNOT ever be `break` as that wouldn't exit the parent job,
+        # only this step in the workflow, blocking as it awaits the next step
+        return true
       else
-        raise UnknownRecoveryPoint unless phases.key? recovery_point
-
-        atomic_phase @key, phases[recovery_point]
+        acidic_step @key, current_step
       end
     end
 
@@ -89,9 +118,14 @@ module AcidicJob
     @key.succeeded?
   end
 
-  def step(method_name)
+  def step(method_name, awaits: [])
     @_steps ||= []
-    @_steps << method_name
+
+    @_steps << {
+      "does" => method_name.to_s,
+      "awaits" => awaits
+    }
+
     @_steps
   end
 
@@ -103,35 +137,30 @@ module AcidicJob
 
   private
 
-  def atomic_phase(key, proc = nil, &block)
-    rescued_error = false
-    phase_callable = (proc || block)
+  def delete_staged_job_record
+    return unless staged_job_gid
 
-    begin
-      key.with_lock do
-        phase_result = phase_callable.call
+    staged_job = GlobalID::Locator.locate(staged_job_gid)
+    staged_job.delete
+    true
+  rescue ActiveRecord::RecordNotFound
+    true
+  end
 
-        phase_result.call(key: key)
-      end
-    rescue StandardError => e
-      rescued_error = e
-      raise e
-    ensure
-      if rescued_error
-        # If we're leaving under an error condition, try to unlock the idempotency
-        # key right away so that another request can try again.3
-        begin
-          key.update_columns(locked_at: nil, error_object: rescued_error)
-        rescue StandardError => e
-          # We're already inside an error condition, so swallow any additional
-          # errors from here and just send them to logs.
-          puts "Failed to unlock key #{key.id} because of #{e}."
-        end
+  def define_workflow(steps)
+    steps << { "does" => Key::RECOVERY_POINT_FINISHED }
+
+    {}.tap do |workflow|
+      steps.each_cons(2).map do |enter_step, exit_step|
+        enter_name = enter_step["does"]
+        workflow[enter_name] = {
+          "then" => exit_step["does"]
+        }.merge(enter_step)
       end
     end
   end
 
-  def ensure_idempotency_key_record(key_val, first_step)
+  def ensure_idempotency_key_record(key_val, workflow, accessors)
     isolation_level = case ActiveRecord::Base.connection.adapter_name.downcase.to_sym
                       when :sqlite
                         :read_uncommitted
@@ -140,29 +169,67 @@ module AcidicJob
                       end
 
     ActiveRecord::Base.transaction(isolation: isolation_level) do
-      @key = Key.find_by(idempotency_key: key_val)
+      key = Key.find_by(idempotency_key: key_val)
 
-      if @key.present?
+      if key.present?
         # Programs enqueuing multiple jobs with different parameters but the
         # same idempotency key is a bug.
-        raise MismatchedIdempotencyKeyAndJobArguments if @key.job_args != @arguments_for_perform
+        raise MismatchedIdempotencyKeyAndJobArguments if key.job_args != @arguments_for_perform
 
         # Only acquire a lock if the key is unlocked or its lock has expired
         # because the original job was long enough ago.
-        raise LockedIdempotencyKey if @key.locked_at && @key.locked_at > Time.current - IDEMPOTENCY_KEY_LOCK_TIMEOUT
+        raise LockedIdempotencyKey if key.locked_at && key.locked_at > Time.current - IDEMPOTENCY_KEY_LOCK_TIMEOUT
 
         # Lock the key and update latest run unless the job is already finished.
-        @key.update!(last_run_at: Time.current, locked_at: Time.current) unless @key.finished?
+        key.update!(last_run_at: Time.current, locked_at: Time.current) unless key.finished?
       else
-        @key = Key.create!(
+        key = Key.create!(
           idempotency_key: key_val,
           locked_at: Time.current,
           last_run_at: Time.current,
-          recovery_point: first_step,
+          recovery_point: workflow.first.first,
           job_name: self.class.name,
-          job_args: @arguments_for_perform
+          job_args: @arguments_for_perform,
+          workflow: workflow
         )
       end
+
+      # set accessors for each argument passed in to ensure they are available
+      # to the step methods the job will have written
+      define_accessors_for_passed_arguments(accessors, key)
+
+      # NOTE: we must return the `key` object from this transaction block
+      # so that it can be returned from this method to the caller
+      key
+    end
+  end
+
+  def acidic_step(key, step)
+    rescued_error = false
+    step_callable = wrap_step_as_acidic_callable step
+
+    begin
+      key.with_lock do
+        step_result = step_callable.call(key)
+
+        step_result.call(key: key)
+      end
+    # QUESTION: Can an error not inherit from StandardError
+    rescue StandardError => e
+      rescued_error = e
+      raise e
+    ensure
+      if rescued_error
+        # If we're leaving under an error condition, try to unlock the idempotency
+        # key right away so that another request can try again.3
+        begin
+          key.update_columns(locked_at: nil, error_object: rescued_error)
+        rescue StandardError => e
+          # We're already inside an error condition, so swallow any additional
+          # errors from here and just send them to logs.
+          puts "Failed to unlock key #{key.id} because of #{e}."
+        end
+      end
     end
   end
 
@@ -192,20 +259,65 @@ module AcidicJob
     true
   end
 
-  def define_atomic_phases(defined_steps)
-    defined_steps << Key::RECOVERY_POINT_FINISHED
-
-    {}.tap do |phases|
-      defined_steps.each_cons(2).map do |enter_method, exit_method|
-        phases[enter_method] = lambda do
-          result = method(enter_method).call
-
-          if result.is_a?(Response)
-            result
-          elsif exit_method.to_s == Key::RECOVERY_POINT_FINISHED
-            Response.new
+  # rubocop:disable Metrics/PerceivedComplexity
+  def wrap_step_as_acidic_callable(step)
+    # {:then=>:next_step, :does=>:enqueue_step, :awaits=>[WorkerWithEnqueueStep::FirstWorker]}
+    current_step = step["does"]
+    next_step = step["then"]
+
+    callable = if respond_to? current_step, _include_private = true
+                 method(current_step)
+               else
+                 proc {} # no-op
+               end
+
+    proc do |key|
+      result = if callable.arity.zero?
+                 callable.call
+               elsif callable.arity == 1
+                 callable.call(key)
+               else
+                 # TODO
+                 raise
+               end
+
+      if result.is_a?(Response)
+        result
+      elsif next_step.to_s == Key::RECOVERY_POINT_FINISHED
+        Response.new
+      else
+        RecoveryPoint.new(next_step)
+      end
+    end
+  end
+  # rubocop:enable Metrics/PerceivedComplexity
+
+  def enqueue_step_parallel_jobs(jobs)
+    # TODO: GIVE PROPER ERROR
+    # `batch` is available from Sidekiq::Pro
+    raise unless defined?(Sidekiq::Batch)
+
+    batch.jobs do
+      step_batch = Sidekiq::Batch.new
+      # step_batch.description = "AcidicJob::Workflow Step: #{step}"
+      step_batch.on(
+        :success,
+        "#{self.class.name}#step_done",
+        # NOTE: options are marshalled through JSON so use only basic types.
+        { "key_id" => @key.id }
+      )
+      # NOTE: The jobs method is atomic.
+      # All jobs created in the block are actually pushed atomically at the end of the block.
+      # If an error is raised, none of the jobs will go to Redis.
+      step_batch.jobs do
+        jobs.each do |worker_name|
+          worker = worker_name.is_a?(String) ? worker_name.constantize : worker_name
+          if worker.instance_method(:perform).arity.zero?
+            worker.perform_async
+          elsif worker.instance_method(:perform).arity == 1
+            worker.perform_async(key.id)
           else
-            RecoveryPoint.new(exit_method)
+            raise
           end
         end
       end
@@ -218,5 +330,11 @@ module AcidicJob
 
     Digest::SHA1.hexdigest [self.class.name, arguments_for_perform].flatten.join
   end
+
+  def step_done(_status, options)
+    key = Key.find(options["key_id"])
+    # when a batch of jobs for a step succeeds, we begin the key processing again
+    process_key(key)
+  end
 end
 # rubocop:enable Metrics/ModuleLength, Metrics/AbcSize, Metrics/MethodLength

data/lib/generators/templates/create_acidic_job_keys_migration.rb.erb

@@ -9,6 +9,7 @@ class CreateAcidicJobKeys < <%= migration_class %>
       t.string :recovery_point, null: false
       t.text :error_object
       t.text :attr_accessors
+      t.text :workflow
       t.timestamps
 
       t.index %i[idempotency_key job_name job_args],

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: acidic_job
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - fractaledmind
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-10-19 00:00:00.000000000 Z
+date: 2021-10-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -73,6 +73,7 @@ files:
 - bin/setup
 - blog_post.md
 - lib/acidic_job.rb
+- lib/acidic_job/deliver_transactionally_extension.rb
 - lib/acidic_job/errors.rb
 - lib/acidic_job/key.rb
 - lib/acidic_job/no_op.rb
@@ -80,12 +81,12 @@ files:
 - lib/acidic_job/perform_wrapper.rb
 - lib/acidic_job/recovery_point.rb
 - lib/acidic_job/response.rb
+- lib/acidic_job/sidekiq_callbacks.rb
 - lib/acidic_job/staged.rb
 - lib/acidic_job/version.rb
 - lib/generators/acidic_job_generator.rb
 - lib/generators/templates/create_acidic_job_keys_migration.rb.erb
 - lib/generators/templates/create_staged_acidic_jobs_migration.rb.erb
-- slides.md
 homepage: https://github.com/fractaledmind/acidic_job
 licenses:
 - MIT

data/slides.md

@@ -1,65 +0,0 @@
-# ACIDic Jobs
-
-## A bit about me
-
-- programming in Ruby for 6 years
-- working for test IO / EPAM
-- consulting for RCRDSHP
-- building Smokestack QA on the side
-
-## Jobs are essential
-
-- job / operation / work
-- in every company, with every app, jobs are essential. Why?
-- jobs are what your app *does*, expressed as a distinct unit
-- jobs can be called from anywhere, run sync or async, and have retry mechanisms built-in
-
-## Jobs are internal API endpoints
-
-- Like API endpoints, both are discrete units of work
-- Like API endpoints, we should expect failure
-- Like API endpoints, we should expect retries
-- Like API endpoints, we should expect concurrency
-- this symmetry allows us to port much of the wisdom built up over decades of building robust APIs to our app job infrastructure
-
-## ACIDic APIs
-
-In a loosely collected series of articles, Brandur Leach lays out the core techniques and principles required to make an HTTP API properly ACIDic:
-
-1. https://brandur.org/acid
-2. https://brandur.org/http-transactions
-3. https://brandur.org/job-drain
-4. https://brandur.org/idempotency-keys
-
-His central points can be summarized as follows:
-
-- "ACID databases are one of the most important tools in existence for ensuring maintainability and data correctness in big production systems"
-- "for a common idempotent HTTP request, requests should map to backend transactions at 1:1"
-- "We can dequeue jobs gracefully by using a transactionally-staged job drain."
-- "Implementations that need to make synchronous changes in foreign state (i.e. outside of a local ACID store) are somewhat more difficult to design. ... To guarantee idempotency on this type of endpoint we’ll need to introduce idempotency keys."
-
-Key concepts:
-
-- foreign state mutations
-  - The reason that the local vs. foreign distinction matters is that unlike a local set of operations where we can leverage an ACID store to roll back a result that we didn’t like, once we make our first foreign state mutation, we’re committed one way or another
-- "An atomic phase is a set of local state mutations that occur in transactions between foreign state mutations."
-- "A recovery point is a name of a check point that we get to after having successfully executed any atomic phase or foreign state mutation"
-- "transactionally-staged job drain"
-  - "With this pattern, jobs aren’t immediately sent to the job queue. Instead, they’re staged in a table within the relational database itself, and the ACID properties of the running transaction keep them invisible until they’re ready to be worked. A secondary enqueuer process reads the table and sends any jobs it finds to the job queue before removing their rows."
-
-
-https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional
-
-2. Make your job idempotent and transactional
-
-Idempotency means that your job can safely execute multiple times. For instance, with the error retry functionality, your job might be half-processed, throw an error, and then be re-executed over and over until it successfully completes. Let's say you have a job which voids a credit card transaction and emails the user to let them know the charge has been refunded:
-
-```ruby
-def perform(card_charge_id)
-  charge = CardCharge.find(card_charge_id)
-  charge.void_transaction
-  Emailer.charge_refunded(charge).deliver
-end
-```
-
-What happens when the email fails to render due to a bug? Will the void_transaction method handle the case where a charge has already been refunded? You can use a database transaction to ensure data changes are rolled back if there is an error or you can write your code to be resilient in the face of errors. Just remember that Sidekiq will execute your job at least once, not exactly once.