good_job 3.8.0 → 3.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d5d0c455a8d5d8f18fda38f1f9f126cc406250d188ee313d0352717c829cd126
-  data.tar.gz: 208b61ec3b48a7973b160093100b216eafbb96eee604e5e7edc2e7fb67bedfba
+  metadata.gz: 22496fe725c6ad616c39f243cf8f53381dee64a5402d3a199c8c604bff2c4eb3
+  data.tar.gz: 6d9791dd28f28ddf9317f2770029323754fa32c60403f336f23db8020cdecc51
 SHA512:
-  metadata.gz: cac4f2c135e6c01cbdd6cd58c591d726ee8fbb66e623e3d1ed8bfb26e85e6b058e45bd60e3d2ba67b3133666df07d972db8eca75aa05df671b8284c01f45398f
-  data.tar.gz: 189ba8af30dc9e5a84c064090d5ede8a8f1da4bb56d2249f6ac4130ce3105c9c498f957ad05cf054f2697f83885570013c3c41ed5d197354c992b08518110656
+  metadata.gz: e182b3aa78b38fd75baa5e0c14704baa99b7aa98f34ff011006543ce7a356e6eae8ac45e605d65d985b8631f6f185666f78e5bb1be1f39feebd62c159535a7e5
+  data.tar.gz: acf8a1f601e07142b0dbaea2f2e229fa3d9b7f8c50dd6232531f8ff50cf888d61f88f2a02ae423ffaa8a9a6533bd1cd0522e7cfa849abb20db8dc538adf75ae3

data/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## [v3.10.0](https://github.com/bensheldon/good_job/tree/v3.10.0) (2023-02-04)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.9.0...v3.10.0)
+
+**Implemented enhancements:**
+
+- Implement `GoodJob::Batch` [\#712](https://github.com/bensheldon/good_job/pull/712) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Support for Ruby 3.2 [\#785](https://github.com/bensheldon/good_job/issues/785)
+- Custom table names [\#748](https://github.com/bensheldon/good_job/issues/748)
+- Health check issue with cron scheduler job [\#741](https://github.com/bensheldon/good_job/issues/741)
+
+## [v3.9.0](https://github.com/bensheldon/good_job/tree/v3.9.0) (2023-01-31)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.8.0...v3.9.0)
+
+**Implemented enhancements:**
+
+- Abort enqueue when the concurrency limit is reached [\#820](https://github.com/bensheldon/good_job/pull/820) ([TAGraves](https://github.com/TAGraves))
+- Add bulk enqueue functionality [\#790](https://github.com/bensheldon/good_job/pull/790) ([julik](https://github.com/julik))
+
+**Merged pull requests:**
+
+- Bump alex-page/github-project-automation-plus from 0.8.2 to 0.8.3 [\#819](https://github.com/bensheldon/good_job/pull/819) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump concurrent-ruby from 1.1.10 to 1.2.0 [\#818](https://github.com/bensheldon/good_job/pull/818) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump rails from 6.1.7 to 6.1.7.2 [\#817](https://github.com/bensheldon/good_job/pull/817) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump selenium-webdriver from 4.7.1 to 4.8.0 [\#816](https://github.com/bensheldon/good_job/pull/816) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump rubocop from 1.43.0 to 1.44.1 [\#815](https://github.com/bensheldon/good_job/pull/815) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Ensure that anytime the Notifier uses autoloaded constants \(ActiveRecord\), they are wrapped with a Rails Executor [\#797](https://github.com/bensheldon/good_job/pull/797) ([bensheldon](https://github.com/bensheldon))
+- Remove support for Ruby 2.5 and JRuby 9.2; reactivate appraisal tests for Rails HEAD [\#756](https://github.com/bensheldon/good_job/pull/756) ([bensheldon](https://github.com/bensheldon))
+
 ## [v3.8.0](https://github.com/bensheldon/good_job/tree/v3.8.0) (2023-01-27)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v3.7.4...v3.8.0)
@@ -68,13 +101,16 @@
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v3.7.1...v3.7.2)
 
+**Fixed bugs:**
+
+- Ignore ActiveJob::DeserializationError when discarding jobs [\#771](https://github.com/bensheldon/good_job/pull/771) ([nickcampbell18](https://github.com/nickcampbell18))
+
 **Closed issues:**
 
 - Unable to discard failed jobs which crashed with `ActiveJob::DeserializationError` [\#770](https://github.com/bensheldon/good_job/issues/770)
 
 **Merged pull requests:**
 
-- Ignore ActiveJob::DeserializationError when discarding jobs [\#771](https://github.com/bensheldon/good_job/pull/771) ([nickcampbell18](https://github.com/nickcampbell18))
 - Bump rubocop from 1.39.0 to 1.40.0 [\#769](https://github.com/bensheldon/good_job/pull/769) ([dependabot[bot]](https://github.com/apps/dependabot))
 
 ## [v3.7.1](https://github.com/bensheldon/good_job/tree/v3.7.1) (2022-12-12)

data/README.md

@@ -41,9 +41,11 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Dashboard](#dashboard)
         - [API-only Rails applications](#api-only-rails-applications)
         - [Live Polling](#live-polling)
-    - [ActiveJob concurrency](#activejob-concurrency)
+    - [Concurrency controls](#concurrency-controls)
         - [How concurrency controls work](#how-concurrency-controls-work)
     - [Cron-style repeating/recurring jobs](#cron-style-repeatingrecurring-jobs)
+    - [Bulk enqueue](#bulk-enqueue)
+    - [Batches](#batches)
     - [Updating](#updating)
         - [Upgrading minor versions](#upgrading-minor-versions)
         - [Upgrading v2 to v3](#upgrading-v2-to-v3)
@@ -146,7 +148,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
 ## Compatibility
 
 - **Ruby on Rails:** 6.0+
-- **Ruby:** Ruby 2.5+. JRuby 9.2.13+
+- **Ruby:** Ruby 2.6+. JRuby 9.3+
 - **Postgres:** 10.0+
 
 ## Configuration
@@ -391,7 +393,7 @@ end
 
 The Dashboard can be set to automatically refresh by checking "Live Poll" in the Dashboard header, or by setting `?poll=10` with the interval in seconds (default 30 seconds).
 
-### ActiveJob concurrency
+### Concurrency controls
 
 GoodJob can extend ActiveJob to provide limits on concurrently running jobs, either at time of _enqueue_ or at _perform_. Limiting concurrency can help prevent duplicate, double or unnecessary jobs from being enqueued, or race conditions when performing, for example when interacting with 3rd-party APIs.
 
@@ -484,6 +486,196 @@ config.good_job.cron = {
 }
 ```
 
+### Bulk enqueue
+
+GoodJob's Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can more performant when enqueuing a large number of jobs.
+
+```ruby
+# Capture jobs using `.perform_later`:
+active_jobs = GoodJob::Bulk.enqueue do
+  MyJob.perform_later
+  AnotherJob.perform_later
+  # If an exception is raised within this block, no jobs will be inserted.
+end
+
+# All ActiveJob instances are returned from GoodJob::Bulk.enqueue.
+# Jobs that have been successfully enqueued have a `provider_job_id` set.
+active_jobs.all?(&:provider_job_id)
+
+# Bulk enqueue ActiveJob instances directly without using `.perform_later`:
+GoodJob::Bulk.enqueue(MyJob.new, AnotherJob.new)
+```
+
+### Batches
+
+Batches track a set of jobs, and enqueue an optional callback job when all of the jobs have finished (succeeded or discarded).
+
+- A simple example that enqueues your `MyBatchCallbackJob` after the two jobs have finished, and passes along the current user as a batch property:
+
+    ```ruby
+    GoodJob::Batch.enqueue(on_finish: MyBatchCallbackJob, user: current_user) do
+      MyJob.perform_later
+      OtherJob.perform_later
+    end
+
+    # When these jobs have finished, it will enqueue your `MyBatchCallbackJob.perform_later(batch, options)`
+    class MyBatchCallbackJob < ApplicationJob
+      # Callback jobs must accept a `batch` and `options` argument
+      def perform(batch, params)
+        # The batch object will contain the Batch's properties, which are mutable
+        batch.properties[:user] # => <User id: 1, ...>
+
+        # Params is a hash containing additional context (more may be added in the future)
+        params[:event] # => :finish, :success, :discard
+      end
+    end
+    ```
+
+- Jobs can be added to an existing batch. Jobs in a batch are enqueued and performed immediately/asynchronously. The final callback job will not be enqueued until `GoodJob::Batch#enqueue` is called.
+
+    ```ruby
+    batch = GoodJob::Batch.add do
+      10.times { MyJob.perform_later }
+    end
+    batch.add do
+      10.times { OtherJob.perform_later }
+    end
+    batch.enqueue(on_finish: MyBatchCallbackJob, age: 42)
+    ```
+
+- If you need to access the batch within a job that is part of the batch, include [`GoodJob::ActiveJobExtensions::Batches`](lib/good_job/active_job_extensions/batches.rb) in your job class:
+
+  ```ruby
+    class MyJob < ApplicationJob
+      include GoodJob::ActiveJobExtensions::Batches
+
+      def perform
+        self.batch # => <GoodJob::Batch id: 1, ...>
+      end
+    end
+    ```
+
+- [`GoodJob::Batch`](app/models/good_job/batch.rb) has a number of assignable attributes and methods:
+
+```ruby
+batch = GoodJob::Batch.new
+batch.description = "My batch"
+batch.on_finish = "MyBatchCallbackJob" # Callback job when all jobs have finished
+batch.on_success = "MyBatchCallbackJob" # Callback job when/if all jobs have succeeded
+batch.on_discard = "MyBatchCallbackJob" # Callback job when the first job in the batch is discarded
+batch.callback_queue_name = "special_queue" # Optional queue for callback jobs, otherwise will defer to job class
+batch.callback_priority = 10 # Optional priority name for callback jobs, otherwise will defer to job class
+batch.properties = { age: 42 } # Custom data and state to attach to the batch
+batch.add do
+  MyJob.perform_later
+end
+batch.enqueue
+
+batch.discarded? # => Boolean
+batch.discarded_at # => <DateTime>
+batch.finished? # => Boolean
+batch.finished_at # => <DateTime>
+batch.succeeded? # => Boolean
+batch.active_jobs # => Array of ActiveJob::Base-inherited jobs that are part of the batch
+
+batch = GoodJob::Batch.find(batch.id)
+batch.description = "Updated batch description"
+batch.save
+batch.reload
+```
+
+### Batch callback jobs
+
+Batch callbacks are Active Job jobs that are enqueued at certain events during the execution of jobs within the batch:
+
+- `:finish` - Enqueued when all jobs in the batch have finished, after all retries. Jobs will either be discarded or succeeded.
+- `:success` - Enqueued only when all jobs in the batch have finished and succeeded.
+- `:discard` - Enqueued immediately the first time a job in the batch is discarded.
+
+Callback jobs must accept a `batch` and `params` argument in their `perform` method:
+
+```ruby
+class MyBatchCallbackJob < ApplicationJob
+  def perform(batch, params)
+    # The batch object will contain the Batch's properties
+    batch.properties[:user] # => <User id: 1, ...>
+    # Batches are mutable
+    batch.properties[:user] = User.find(2)
+    batch.save
+
+    # Params is a hash containing additional context (more may be added in the future)
+    params[:event] # => :finish, :success, :discard
+  end
+end
+```
+
+#### Complex batches
+
+Consider a multi-stage batch with both parallel and serial job steps:
+
+```mermaid
+graph TD
+    0{"BatchJob\n{ stage: nil }"}
+    0 --> a["WorkJob]\n{ step: a }"]
+    0 --> b["WorkJob]\n{ step: b }"]
+    0 --> c["WorkJob]\n{ step: c }"]
+    a --> 1
+    b --> 1
+    c --> 1
+    1{"BatchJob\n{ stage: 1 }"}
+    1 --> d["WorkJob]\n{ step: d }"]
+    1 --> e["WorkJob]\n{ step: e }"]
+    e --> f["WorkJob]\n{ step: f }"]
+    d --> 2
+    f --> 2
+    2{"BatchJob\n{ stage: 2 }"}
+```
+
+This can be implemented with a single, mutable batch job:
+
+```ruby
+class WorkJob < ApplicationJob
+  include GoodJob::ActiveJobExtensions::Batches
+
+  def perform(step)
+    # ...
+    if step == 'e'
+      batch.add { WorkJob.perform_later('f') }
+    end
+  end
+end
+
+class BatchJob < ApplicationJob
+  def perform(batch, options)
+    if batch.properties[:stage].nil?
+      batch.enqueue(stage: 1) do
+        WorkJob.perform_later('a')
+        WorkJob.perform_later('b')
+        WorkJob.perform_later('c')
+      end
+    elsif batch.properties[:stage] == 1
+      batch.enqueue(stage: 2) do
+        WorkJob.perform_later('d')
+        WorkJob.perform_later('e')
+      end
+    elsif batch.properties[:stage] == 2
+      # ...
+    end
+  end
+end
+
+GoodJob::Batch.enqueue(on_finish: BatchJob)
+```
+
+#### Other batch details
+
+- Whether to enqueue a callback job is evaluated once the batch is in an `enqueued?`-state by using `GoodJob::Batch.enqueue` or `batch.enqueue`.
+- Callback job enqueueing will be re-triggered if additional jobs are `enqueue`'d to the batch; use `add` to add jobs to the batch without retriggering callback jobs.
+- Callback jobs will be enqueued even if the batch contains no jobs.
+- Callback jobs perform asynchronously. It's possible that `:finish` and `:success` or `:discard` callback jobs perform at the same time. Keep this in mind when updating batch properties.
+- Batch properties are serialized using Active Job serialization. This is flexible, but can lead to deserialization errors if a GlobalID record is directly referenced but is subsequently deleted and thus unloadable.
+- 🚧Batches are a work in progress. Please let us know what would be helpful to improve their functionality and usefulness.
+
 ### Updating
 
 GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.

data/app/controllers/good_job/batches_controller.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+module GoodJob
+  class BatchesController < GoodJob::ApplicationController
+    def index
+      @filter = BatchesFilter.new(params)
+    end
+
+    def show
+      @batch = GoodJob::BatchRecord.find(params[:id])
+    end
+  end
+end

data/app/filters/good_job/batches_filter.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+module GoodJob
+  class BatchesFilter < BaseFilter
+    def records
+      after_created_at = params[:after_created_at].present? ? Time.zone.parse(params[:after_created_at]) : nil
+
+      filtered_query.display_all(
+        after_created_at: after_created_at,
+        after_id: params[:after_id]
+      ).limit(params.fetch(:limit, DEFAULT_LIMIT))
+    end
+
+    def filtered_query(_filtered_params = params)
+      base_query
+    end
+
+    def default_base_query
+      GoodJob::BatchRecord.all.includes(:jobs)
+    end
+  end
+end

data/app/models/good_job/batch.rb

@@ -0,0 +1,141 @@
+# frozen_string_literal: true
+
+module GoodJob
+  # NOTE: This class delegates to {GoodJob::BatchRecord} and is intended to be the public interface for Batches.
+  class Batch
+    include GlobalID::Identification
+
+    thread_cattr_accessor :current_batch_id
+    thread_cattr_accessor :current_batch_callback_id
+
+    PROTECTED_PROPERTIES = %i[
+      on_finish
+      on_success
+      on_discard
+      callback_queue_name
+      callback_priority
+      description
+      properties
+    ].freeze
+
+    delegate(
+      :id,
+      :created_at,
+      :updated_at,
+      :persisted?,
+      :enqueued_at,
+      :finished_at,
+      :discarded_at,
+      :enqueued?,
+      :finished?,
+      :succeeded?,
+      :discarded?,
+      :description,
+      :description=,
+      :on_finish,
+      :on_finish=,
+      :on_success,
+      :on_success=,
+      :on_discard,
+      :on_discard=,
+      :callback_queue_name,
+      :callback_queue_name=,
+      :callback_priority,
+      :callback_priority=,
+      :properties,
+      :properties=,
+      :save,
+      :reload,
+      to: :record
+    )
+
+    # Create a new batch and enqueue it
+    # @param on_finish [String, Object] The class name of the callback job to be enqueued after the batch is finished
+    # @param properties [Hash] Additional properties to be stored on the batch
+    # @param block [Proc] Enqueue jobs within the block to add them to the batch
+    # @return [GoodJob::BatchRecord]
+    def self.enqueue(active_jobs = [], **properties, &block)
+      new.tap do |batch|
+        batch.enqueue(active_jobs, **properties, &block)
+      end
+    end
+
+    def self.find(id)
+      new _record: BatchRecord.find(id)
+    end
+
+    # Helper method to enqueue jobs and assign them to a batch
+    def self.within_thread(batch_id: nil, batch_callback_id: nil)
+      original_batch_id = current_batch_id
+      original_batch_callback_id = current_batch_callback_id
+
+      self.current_batch_id = batch_id
+      self.current_batch_callback_id = batch_callback_id
+
+      yield
+    ensure
+      self.current_batch_id = original_batch_id
+      self.current_batch_callback_id = original_batch_callback_id
+    end
+
+    def initialize(_record: nil, **properties) # rubocop:disable Lint/UnderscorePrefixedVariableName
+      self.record = _record || BatchRecord.new
+      assign_properties(properties)
+    end
+
+    # @return [Array<ActiveJob::Base>] Active jobs added to the batch
+    def enqueue(active_jobs = [], **properties, &block)
+      assign_properties(properties)
+      record.save!
+
+      active_jobs = add(active_jobs, &block)
+
+      record.finished_at = nil
+      record.enqueued_at = Time.current if enqueued_at.nil?
+      record.save!
+
+      record._continue_discard_or_finish
+      active_jobs
+    end
+
+    # Enqueue jobs and add them to the batch
+    # @param block [Proc] Enqueue jobs within the block to add them to the batch
+    # @return [Array<ActiveJob::Base>] Active jobs added to the batch
+    def add(active_jobs = nil, &block)
+      record.save if record.new_record?
+
+      buffer = Bulk::Buffer.new
+      buffer.add(active_jobs)
+      buffer.capture(&block) if block
+
+      self.class.within_thread(batch_id: id) do
+        buffer.enqueue
+      end
+
+      buffer.active_jobs
+    end
+
+    def active_jobs
+      record.jobs.map(&:head_execution).map(&:active_job)
+    end
+
+    def callback_active_jobs
+      record.callback_jobs.map(&:head_execution).map(&:active_job)
+    end
+
+    def assign_properties(properties)
+      properties = properties.dup
+      batch_attrs = PROTECTED_PROPERTIES.index_with { |key| properties.delete(key) }.compact
+      record.assign_attributes(batch_attrs)
+      self.properties.merge!(properties)
+    end
+
+    def _record
+      record
+    end
+
+    private
+
+    attr_accessor :record
+  end
+end

data/app/models/good_job/batch_record.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+module GoodJob
+  class BatchRecord < BaseRecord
+    include Lockable
+
+    self.table_name = 'good_job_batches'
+
+    has_many :jobs, class_name: 'GoodJob::Job', inverse_of: :batch, foreign_key: :batch_id, dependent: nil
+    has_many :executions, class_name: 'GoodJob::Execution', foreign_key: :batch_id, inverse_of: :batch, dependent: nil
+    has_many :callback_jobs, class_name: 'GoodJob::Job', foreign_key: :batch_callback_id, dependent: nil # rubocop:disable Rails/InverseOf
+
+    scope :finished, -> { where.not(finished_at: nil) }
+    scope :discarded, -> { where.not(discarded_at: nil) }
+    scope :not_discarded, -> { where(discarded_at: nil) }
+    scope :succeeded, -> { finished.not_discarded }
+
+    alias_attribute :enqueued?, :enqueued_at
+    alias_attribute :discarded?, :discarded_at
+    alias_attribute :finished?, :finished_at
+
+    scope :display_all, (lambda do |after_created_at: nil, after_id: nil|
+      query = order(created_at: :desc, id: :desc)
+      if after_created_at.present? && after_id.present?
+        query = query.where(Arel.sql('(created_at, id) < (:after_created_at, :after_id)'), after_created_at: after_created_at, after_id: after_id)
+      elsif after_created_at.present?
+        query = query.where(Arel.sql('(after_created_at) < (:after_created_at)'), after_created_at: after_created_at)
+      end
+      query
+    end)
+
+    # Whether the batch has finished and no jobs were discarded
+    # @return [Boolean]
+    def succeeded?
+      !discarded? && finished?
+    end
+
+    def to_batch
+      Batch.new(_record: self)
+    end
+
+    def display_attributes
+      attributes.except('serialized_properties').merge(properties: properties)
+    end
+
+    def _continue_discard_or_finish(execution = nil)
+      execution_discarded = execution && execution.error.present? && execution.retried_good_job_id.nil?
+      with_advisory_lock(function: "pg_advisory_lock") do
+        Batch.within_thread(batch_id: nil, batch_callback_id: id) do
+          if execution_discarded && discarded_at.blank?
+            update(discarded_at: Time.current)
+            on_discard.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :discard }) if on_discard.present?
+          end
+
+          if !finished_at && enqueued_at && jobs.where(finished_at: nil).count.zero?
+            update(finished_at: Time.current)
+            on_success.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :success }) if !discarded_at && on_success.present?
+            on_finish.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :finish }) if on_finish.present?
+          end
+        end
+      end
+    end
+
+    class PropertySerializer
+      def self.dump(value)
+        ActiveJob::Arguments.serialize([value]).first
+      end
+
+      def self.load(value)
+        ActiveJob::Arguments.deserialize([value]).first
+      end
+    end
+
+    attribute :serialized_properties, :json, default: -> { {} } # Can be set as default value in `serialize` as of Rails v6.1
+    serialize :serialized_properties, PropertySerializer
+    alias_attribute :properties, :serialized_properties
+
+    def properties=(value)
+      raise ArgumentError, "Properties must be a Hash" unless value.is_a?(Hash)
+
+      self.serialized_properties = value
+    end
+  end
+end