acidic_job 1.0.0.pre29 → 1.0.0.rc2

data/TODO

@@ -0,0 +1,77 @@
+- [x] Track number of runs per execution (resolves #79)
+  This is accomplished with the new `entries` table, as proven by the `idempotency_check` feature, which only runs on a retry.
+
+- [x] Store errors with more context and without full error history (resolves #80)
+  This is also accomplished with the new `entries` table, by storing a separate error entry for any raised errors.
+
+- [x] All traversal over a collection added to the context within the workflow (resolves #81)
+  The new collection traversal logic resolves #81, as proven by the test case. You can traverse a key for a context value. For example,
+  ```ruby
+  execute :prepare_collection
+  traverse :collection, using: :process_item
+
+  def prepare_collection = @ctx[:collection] = 1..5
+  def process_item(item) = @processed_items << item
+  ```
+
+- [x] Handle case where non-idempotent external write succeeds but recording fails (resolves #82)
+  The new `idempotency_check` option accomplishes this, by allowing a step making external IO to define a method that checks if the write was already successfully made when re-running a step in a retry.
+
+- [x] Ensure any and all exceptions can be serialized and deserialized (resolves #83)
+  The original serializer was too naive and couldn't deserialize, for example, an `ActionView::Template::Error` which requires that `$!` be set when initializing the object.
+
+  The new `ExceptionSerializer` uses Ruby's built-in YAML encoding of exceptions to serialize and deserialize the objects, laying on Zlib to ensure compact byte storage.
+
+- [x] Handle duplicate workflow step names (resolves #87)
+  Adding a `DuplicateStepError` exception raised when building the workflow definition resolved this issue.
+
+- [ ] Add a `context` step type (resolves #89)
+  Use case: the workflow may or may not have the data it needs to proceed. If it doesn't, it should make an API call to fetch the data. The API call may take a while, so we should be able to handle that as well.
+
+  Example usage:
+  ```ruby
+  context :slack_author, fallback: :fetch_slack_author
+
+  def slack_author
+    Slack::Profile.find_by(uid: @user_uid)
+  end
+
+  def fetch_slack_author
+    api_response = Slack::Client.new().get_user(uid: @user_uid)
+    if api_response[:ok]
+      [Slack::ProcessProfileSlackJob.new(@installation, api_response[:user])]
+    else
+      raise DoNotRetryJob
+    end
+  end
+  ```
+
+  Psuedo code for implementation:
+  ```ruby
+  result = resolve_method(primary_method_name).call
+  if result
+    @ctx[primary_method_name] = result
+  else
+    fallback = resolve_method(fallback_method_name).call
+    case fallback
+    in ActiveJob::Base then awaits(fallback)
+    in Array[ActiveJob::Base] then awaits(fallback)
+    else @ctx[primary_method_name] = result
+    end
+  end
+  ```
+
+- [ ] Add documentation on how to "migrate" a workflow (resolves #90)
+  The v1 alpha already includes a check that the persisted workflow definition matches the job's current definition. This ensures that no job tries to run with an outdated definition. However, we should document the proper way to update a workflow definition.
+
+  The process can only be additive, like a strong migration. First, you create a new job with a new name that is a clone of the original job. Make the necessary changes to the new job. Update your codebase to only enqueue the new job. Deploy this change, where both the new job and the old jobs exist, but the application only enqueues the new job. Once deployed, wait until all currently running instances of the old job complete (provide some docs on how to check this). Once all old job instances are complete, you can safely delete the old job and deploy that change. This process ensures that no job is running with an outdated definition.
+
+- [ ] Automatically retry the serializable transaction for find/create the Run record (resolves #91)
+  Serializable transactions are prone to failure: https://stackoverflow.com/a/21715207/2884386
+
+  Currently, if the transaction fails, the job will fail. We need to build automatic retries into the gem, as this failure will naturally resolve. But, we should also add a limit to the number of retries to prevent infinite loops.
+
+- [ ] Ensure the gem works with GoodJob (resolves #92 and #94)
+  In the current version, GoodJob can't handle messing with the `job_id` and retrying a failed job raises a NoMethodError: "undefined method `utc' for an instance of Float".
+
+- [ ] Ensure users can transactionally enqueue other entities, like ActionMailer or Noticed objects

data/acidic_job.gemspec

@@ -27,24 +27,24 @@ Gem::Specification.new do |spec|
   spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_dependency "activerecord"
-  spec.add_dependency "activesupport"
-  spec.add_dependency "database_cleaner"
-  spec.add_development_dependency "activejob"
+  spec.add_dependency "json", ">= 2.7.0" # see: https://github.com/ruby/json/pull/519
+  ">= 7.1".tap do |rails_version|
+    spec.add_dependency "activejob", rails_version
+    spec.add_dependency "activerecord", rails_version
+    spec.add_dependency "activesupport", rails_version
+    spec.add_dependency "railties", rails_version
+    spec.add_development_dependency "actionmailer", rails_version
+  end
+
+  spec.add_development_dependency "chaotic_job"
   spec.add_development_dependency "combustion"
   spec.add_development_dependency "minitest"
-  spec.add_development_dependency "net-smtp"
-  spec.add_development_dependency "noticed"
-  spec.add_development_dependency "psych", "> 4.0"
-  spec.add_development_dependency "railties"
   spec.add_development_dependency "rake"
   spec.add_development_dependency "rubocop"
   spec.add_development_dependency "rubocop-minitest"
   spec.add_development_dependency "rubocop-rake"
-  spec.add_development_dependency "sidekiq"
   spec.add_development_dependency "simplecov"
   spec.add_development_dependency "sqlite3"
-  spec.add_development_dependency "warning"
 
   # For more information and examples about making a new gem, checkout our
   # guide at: https://bundler.io/guides/creating_gem.html

data/app/models/acidic_job/entry.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Entry < Record
+    belongs_to :execution, class_name: "AcidicJob::Execution"
+
+    def started?
+      action == "started"
+    end
+
+    def succeeded?
+      action == "succeeded"
+    end
+
+    def errored?
+      action == "errored"
+    end
+  end
+end

data/app/models/acidic_job/execution.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Execution < Record
+    has_many :entries, class_name: "AcidicJob::Entry"
+    has_many :values, class_name: "AcidicJob::Value"
+
+    validates :idempotency_key, presence: true # uniqueness constraint is enforced at the database level
+    validates :serialized_job, presence: true
+
+    scope :finished, -> { where(recover_to: FINISHED_RECOVERY_POINT) }
+    scope :outstanding, lambda {
+                          where.not(recover_to: FINISHED_RECOVERY_POINT).or(where(recover_to: [nil, ""]))
+                        }
+
+    def record!(step:, action:, timestamp:, **kwargs)
+      AcidicJob.instrument(:record_entry, step: step, action: action, timestamp: timestamp, data: kwargs) do
+        entries.create!(
+          step: step,
+          action: action,
+          timestamp: timestamp,
+          data: kwargs.stringify_keys!
+        )
+      end
+    end
+
+    def context
+      @context ||= Context.new(self)
+    end
+
+    def finished?
+      recover_to.to_s == FINISHED_RECOVERY_POINT
+    end
+
+    def deserialized_job
+      serialized_job["job_class"].constantize.new.tap do |job|
+        job.deserialize(serialized_job)
+      end
+    end
+
+    def raw_arguments
+      JSON.parse(serialized_job_before_type_cast)["arguments"]
+    end
+
+    def enqueue_job
+      deserialized_job.enqueue
+      true
+    end
+  end
+end

data/app/models/acidic_job/record.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Record < ActiveRecord::Base
+    self.abstract_class = true
+
+    connects_to(**AcidicJob.connects_to) if AcidicJob.connects_to
+  end
+end
+
+ActiveSupport.run_load_hooks :acidic_job_record, AcidicJob::Record

data/app/models/acidic_job/value.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Value < Record
+    belongs_to :execution, class_name: "AcidicJob::Execution"
+  end
+end

data/bin/console

@@ -2,13 +2,16 @@
 # frozen_string_literal: true
 
 require "bundler/setup"
+require "rails"
 require "acidic_job"
 
 # You can add fixtures and/or initialization code here to make experimenting
 # with your gem easier. You can also use a different console, if you like.
 
-require_relative "../test/support/setup"
-require_relative "../test/support/ride_create_job"
+require "combustion"
+require "sqlite3"
+Combustion.path = "test/combustion"
+Combustion.initialize! :active_record
 
 # (If you use this, don't forget to add pry to your Gemfile!)
 # require "pry"

data/bin/test_all

@@ -0,0 +1,26 @@
+run_tests() {
+  if [ "$#" -ne 2 ]; then
+    echo "Usage: run_tests RUBY_VERSION RAILS_VERSION"
+    echo "Example: run_tests 3.0.7 7.1"
+    return 1
+  fi
+
+  local ruby="$1"
+  local gemfile="gemfiles/rails_$2.gemfile"
+
+  echo "**************************************************"
+  echo "Running tests with Ruby $1 and Rails $2..."
+  ASDF_RUBY_VERSION=$ruby BUNDLE_GEMFILE=$gemfile bundle check ||
+  ASDF_RUBY_VERSION=$ruby BUNDLE_GEMFILE=$gemfile bundle install &&
+  ASDF_RUBY_VERSION=$ruby BUNDLE_GEMFILE=$gemfile bundle exec rake
+}
+
+run_tests "3.0.7" "7.1"
+run_tests "3.1.6" "7.1"
+run_tests "3.1.6" "7.2"
+run_tests "3.2.5" "7.1"
+run_tests "3.2.5" "7.2"
+run_tests "3.2.5" "8.0"
+run_tests "3.3.5" "7.1"
+run_tests "3.3.5" "7.2"
+run_tests "3.3.5" "8.0"

data/gemfiles/rails_7.0.gemfile

@@ -2,7 +2,10 @@
 
 source "https://rubygems.org"
 
-gem "activemodel", "~> 7.0.0"
+gem "activejob", "~> 7.0.0"
+gem "activerecord", "~> 7.0.0"
+gem "activesupport", "~> 7.0.0"
 gem "railties", "~> 7.0.0"
+gem "sqlite3", "~> 1.7.3"
 
 gemspec path: "../"

data/gemfiles/rails_7.1.gemfile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gem "activejob", "~> 7.1.0"
+gem "activerecord", "~> 7.1.0"
+gem "activesupport", "~> 7.1.0"
+gem "railties", "~> 7.1.0"
+gem "sqlite3", ">= 2.0.0"
+
+gemspec path: "../"

data/gemfiles/rails_7.2.gemfile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gem "activejob", "~> 7.2.0"
+gem "activerecord", "~> 7.2.0"
+gem "activesupport", "~> 7.2.0"
+gem "railties", "~> 7.2.0"
+gem "sqlite3", ">= 2.0.0"
+
+gemspec path: "../"

data/gemfiles/rails_8.0.gemfile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gem "activejob", "~> 8.0.0.rc1"
+gem "activerecord", "~> 8.0.0.rc1"
+gem "activesupport", "~> 8.0.0.rc1"
+gem "railties", "~> 8.0.0.rc1"
+gem "sqlite3", ">= 2.1.0"
+
+gemspec path: "../"

data/lib/acidic_job/arguments.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require "active_job/arguments"
+
+module AcidicJob
+  module Arguments
+    include ::ActiveJob::Arguments
+
+    extend self
+
+    # `ActiveJob` will throw an error if it tries to deserialize a GlobalID record.
+    # However, this isn't the behavior that we want for our custom `ActiveRecord` serializer.
+    # Since `ActiveRecord` does _not_ reset instance record state to its pre-transactional state
+    # on a transaction ROLLBACK, we can have GlobalID entries in a serialized column that point to
+    # non-persisted records. This is ok. We should simply return `nil` for that portion of the
+    # serialized field.
+    def deserialize_global_id(hash)
+      ::GlobalID::Locator.locate hash[GLOBALID_KEY]
+    rescue ::ActiveRecord::RecordNotFound
+      nil
+    end
+
+    # In order to allow our `NewRecordSerializer` a chance to work, we need to ensure that
+    # ActiveJob's first attempt to serialize an ActiveRecord model doesn't throw an exception.
+    def convert_to_global_id_hash(argument)
+      { GLOBALID_KEY => argument.to_global_id.to_s }
+    rescue ::URI::GID::MissingModelIdError
+      ::ActiveJob::Serializers.serialize(argument)
+    end
+  end
+end

data/lib/acidic_job/builder.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Builder
+    attr_reader :steps
+
+    def initialize
+      @steps = []
+    end
+
+    def step(method_name, transactional: false)
+      @steps << { "does" => method_name.to_s, "transactional" => transactional }
+      @steps
+    end
+
+    def define_workflow
+      # [ { does: "step 1", transactional: true }, { does: "step 2", transactional: false }, ... ]
+      @steps << { "does" => FINISHED_RECOVERY_POINT }
+
+      {}.tap do |workflow|
+        @steps.each_cons(2).map do |enter_step, exit_step|
+          enter_name = enter_step["does"]
+          workflow[enter_name] = enter_step.merge("then" => exit_step["does"])
+        end
+      end
+      # { "step 1": { does: "step 1", transactional: true, then: "step 2" }, ...  }
+    end
+  end
+end

data/lib/acidic_job/context.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Context
+    def initialize(execution)
+      @execution = execution
+    end
+
+    def set(hash)
+      AcidicJob.instrument(:set_context, **hash) do
+        AcidicJob::Value.upsert_all(
+          hash.map do |key, value|
+            { execution_id: @execution.id,
+              key: key,
+              value: value }
+          end,
+          unique_by: %i[execution_id key]
+        )
+      end
+    end
+
+    def get(*keys)
+      AcidicJob.instrument(:get_context, keys: keys) do
+        @execution.values.select(:value).where(key: keys).pluck(:value)
+      end
+    end
+
+    # TODO: deprecate these methods
+    def []=(key, value)
+      AcidicJob.instrument(:set_context, key: key, value: value) do
+        AcidicJob::Value.upsert(
+          { execution_id: @execution.id,
+            key: key,
+            value: value },
+          unique_by: %i[execution_id key]
+        )
+      end
+    end
+
+    def [](key)
+      AcidicJob.instrument(:get_context, key: key) do
+        @execution.values.select(:value).find_by(key: key)&.value
+      end
+    end
+  end
+end

data/lib/acidic_job/engine.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module AcidicJob
+  class Engine < ::Rails::Engine
+    isolate_namespace AcidicJob
+
+    config.acidic_job = ActiveSupport::OrderedOptions.new
+
+    initializer "acidic_job.config" do
+      config.acidic_job.each do |name, value|
+        AcidicJob.public_send("#{name}=", value)
+      end
+    end
+
+    initializer "acidic_job.logger" do
+      ActiveSupport.on_load :acidic_job do
+        self.logger = ::Rails.logger if logger == AcidicJob::DEFAULT_LOGGER
+      end
+
+      AcidicJob::LogSubscriber.attach_to :acidic_job
+    end
+
+    initializer "acidic_job.active_job.extensions" do
+      ActiveSupport.on_load :active_job do
+        require "active_job/serializers"
+        require_relative "serializers/exception_serializer"
+        require_relative "serializers/new_record_serializer"
+        require_relative "serializers/job_serializer"
+        require_relative "serializers/range_serializer"
+
+        ActiveJob::Serializers.add_serializers(
+          Serializers::ExceptionSerializer,
+          Serializers::NewRecordSerializer,
+          Serializers::JobSerializer,
+          Serializers::RangeSerializer
+        )
+      end
+    end
+
+    # :nocov:
+    generators do
+      require "generators/acidic_job/install_generator"
+    end
+    # :nocov:
+  end
+end

data/lib/acidic_job/errors.rb

@@ -1,27 +1,102 @@
 # frozen_string_literal: true
 
 module AcidicJob
-  class Error < StandardError; end
+  class Error < StandardError
+  end
 
-  class MismatchedIdempotencyKeyAndJobArguments < Error; end
+  class RedefiningWorkflowError < Error
+    def message
+      "can only call `execute_workflow` once within a job"
+    end
+  end
 
-  class LockedIdempotencyKey < Error; end
+  class UndefinedWorkflowBlockError < Error
+    def message
+      "block must be passed to `execute_workflow`"
+    end
+  end
 
-  class UnknownRecoveryPoint < Error; end
+  class InvalidWorkflowBlockError < Error
+    def message
+      "workflow builder must be yielded to the `execute_workflow` block"
+    end
+  end
 
-  class UnknownAtomicPhaseType < Error; end
+  class MissingStepsError < Error
+    def message
+      "workflow must define at least one step"
+    end
+  end
 
-  class SerializedTransactionConflict < Error; end
+  # rubocop:disable Lint/MissingSuper
+  class ArgumentMismatchError < Error
+    def initialize(expected, existing)
+      @expected = expected
+      @existing = existing
+    end
 
-  class UnknownJobAdapter < Error; end
+    def message
+      <<~TXT
+        existing execution's arguments do not match
+          existing: #{@existing.inspect}
+          expected: #{@expected.inspect}
+      TXT
+    end
+  end
 
-  class NoDefinedSteps < Error; end
+  class DefinitionMismatchError < Error
+    def initialize(expected, existing)
+      @expected = expected
+      @existing = existing
+    end
 
-  class SidekiqBatchRequired < Error; end
+    def message
+      <<~TXT
+        existing execution's definition does not match
+          existing: #{@existing.inspect}
+          expected: #{@expected.inspect}
+      TXT
+    end
+  end
 
-  class TooManyParametersForStepMethod < Error; end
+  class UndefinedStepError < Error
+    def initialize(step)
+      @step = step
+    end
 
-  class TooManyParametersForParallelJob < Error; end
+    def message
+      "workflow does not reference this step: #{@step.inspect}"
+    end
+  end
 
-  class UnknownSerializedJobIdentifier < Error; end
+  class SucceededStepError < Error
+    def initialize(step)
+      @step = step
+    end
+
+    def message
+      "workflow has already recorded this step as succeeded: #{@step.inspect}"
+    end
+  end
+
+  class UndefinedMethodError < Error
+    def initialize(step)
+      @step = step
+    end
+
+    def message
+      "Undefined step method: #{@step.inspect}"
+    end
+  end
+
+  class InvalidMethodError < Error
+    def initialize(step)
+      @step = step
+    end
+
+    def message
+      "step method cannot expect arguments: #{@step.inspect}"
+    end
+  end
+  # rubocop:enable Lint/MissingSuper
 end

data/lib/acidic_job/log_subscriber.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require "active_support/log_subscriber"
+
+module AcidicJob
+  class LogSubscriber < ActiveSupport::LogSubscriber
+    def define_workflow(event)
+      debug formatted_event(event, action: "Define workflow", **event.payload.slice("job_class", "job_id"))
+    end
+
+    def initialize_workflow(event)
+      debug formatted_event(event, action: "Initialize workflow", **event.payload.slice("steps"))
+    end
+
+    def process_workflow(event)
+      debug formatted_event(event, action: "Process workflow", **event.payload["execution"].slice("id", "recover_to"))
+    end
+
+    def process_step(event)
+      debug formatted_event(event, action: "Process step", **event.payload)
+    end
+
+    def perform_step(event)
+      debug formatted_event(event, action: "Perform step", **event.payload)
+    end
+
+    def record_entry(event)
+      debug formatted_event(event, action: "Record entry", **event.payload.slice(:step, :action, :timestamp))
+    end
+
+    private
+
+    def formatted_event(event, action:, **attributes)
+      "AcidicJob-#{AcidicJob::VERSION} #{action} (#{event.duration.round(1)}ms)  #{formatted_attributes(**attributes)}"
+    end
+
+    def formatted_attributes(**attributes)
+      attributes.map { |attr, value| "#{attr}: #{value.inspect}" }.join(", ")
+    end
+
+    def formatted_error(error)
+      [error.class, error.message].compact.join(" ")
+    end
+
+    # Use the logger configured for AcidicJob
+    def logger
+      AcidicJob.logger
+    end
+  end
+end

data/lib/acidic_job/serializers/exception_serializer.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require "active_job/serializers/object_serializer"
+require "zlib"
+require "yaml"
+
+module AcidicJob
+  module Serializers
+    class ExceptionSerializer < ::ActiveJob::Serializers::ObjectSerializer
+      def serialize(exception)
+        compressed = Zlib::Deflate.deflate(exception.to_yaml)
+
+        super("deflated_yaml" => compressed)
+      end
+
+      def deserialize(hash)
+        uncompressed = Zlib::Inflate.inflate(hash["deflated_yaml"])
+
+        if YAML.respond_to?(:unsafe_load)
+          YAML.unsafe_load(uncompressed)
+        else
+          YAML.load(uncompressed) # rubocop:disable Security/YAMLLoad
+        end
+      end
+
+      def serialize?(argument)
+        defined?(Exception) && argument.is_a?(Exception)
+      end
+    end
+  end
+end