good_job 3.5.1 → 3.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1d98ed0a03595d8d2a70f2a811296cf01d1b0a4d9ff480d0a480fb91591ecafd
-  data.tar.gz: 90307d30a72de7f7af5d33a7d64ec9eaa67bb42fc8e8179e48c96f22a0440b5e
+  metadata.gz: a3932cd8e0d33ab7f8a56a43c9c458f69db206291b092d9fcac16dfef11849ad
+  data.tar.gz: 695e46da59becc3450c0a0e33c1557784cef56794db979a648dce5d792c57651
 SHA512:
-  metadata.gz: 710500b2704938b319863efe46d462296afa19f11177fb7e03f64e3cfa32872b1d1c8e793a58e6561e0f15a36125b4c3485be99edf6d1e2f8d9953da408ecb3b
-  data.tar.gz: 6d0ae65d53a40340e846cd393a3936fe6ece5b7f5e2ccfdf17ee99f628f528135af11bb2714ee223f1e800b4ced333acc7616a153774d85d9889a0e17753a162
+  metadata.gz: 0ce689ec771af8a472d964cf14739ecc1fd2bf5c903692fbb2d73aeb19870560919c42b9aff5a0ca1671068ad6d914ceaa503aab84649e281ac53c5a17f3a6b5
+  data.tar.gz: 8c85d21ef4dbe48400c24dffa86f7ea78a5ae827ce8057cd8ab2efc439a980c24b61ae8224e14153a0a23b5933d49d8bf7c6e6d0e269b7cdeb7f28ced476b509

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## [v3.6.0](https://github.com/bensheldon/good_job/tree/v3.6.0) (2022-10-22)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.5.1...v3.6.0)
+
+**Fixed bugs:**
+
+- 3.4.8 release breaks job retrying [\#728](https://github.com/bensheldon/good_job/issues/728)
+
+**Merged pull requests:**
+
+- Redo: When not preserving job records, ensure all prior executions are deleted after successful retry [\#730](https://github.com/bensheldon/good_job/pull/730) ([bensheldon](https://github.com/bensheldon))
+- Add configurable limit \(`queue_select_limit`\) when querying candidate jobs [\#727](https://github.com/bensheldon/good_job/pull/727) ([mitchellhenke](https://github.com/mitchellhenke))
+- Add index to `good_jobs` to improve querying candidate jobs [\#726](https://github.com/bensheldon/good_job/pull/726) ([mitchellhenke](https://github.com/mitchellhenke))
+
 ## [v3.5.1](https://github.com/bensheldon/good_job/tree/v3.5.1) (2022-10-20)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v3.5.0...v3.5.1)

data/README.md

@@ -57,6 +57,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Optimize queues, threads, and processes](#optimize-queues-threads-and-processes)
     - [Database connections](#database-connections)
         - [Production setup](#production-setup)
+        - [Queue performance with Queue Select Limit](#queue-performance-with-queue-select-limit)
     - [Execute jobs async / in-process](#execute-jobs-async--in-process)
     - [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
     - [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
@@ -176,6 +177,7 @@ Options:
   [--daemonize]                # Run as a background daemon (default: false)
   [--pidfile=PIDFILE]          # Path to write daemonized Process ID (env var: GOOD_JOB_PIDFILE, default: tmp/pids/good_job.pid)
   [--probe-port=PORT]          # Port for http health check (env var: GOOD_JOB_PROBE_PORT, default: nil)
+  [--queue-select-limit=COUNT] # The number of queued jobs to select when polling for a job to run. (env var: GOOD_JOB_QUEUE_SELECT_LIMIT, default: nil)"
 
 Executes queued jobs.
 
@@ -757,6 +759,45 @@ The recommended way to monitor the queue in production is:
 - keep an eye on the number of jobs in the queue (abnormal high number of unscheduled jobs means the queue could be underperforming)
 - consider performance monitoring services which support the built-in Rails instrumentation (eg. Sentry, Skylight, etc.)
 
+#### Queue performance with Queue Select Limit
+
+GoodJob’s advisory locking strategy uses a materialized CTE (Common Table Expression). This strategy can be non-performant when querying a very large queue of executable jobs (100,000+) because the database query must materialize all executable jobs before acquiring an advisory lock.
+
+GoodJob offers an optional optimization to limit the number of jobs that are queried: Queue Select Limit.
+
+```none
+# CLI option
+--queue-select-limit=1000
+
+# Rails configuration
+config.good_job.queue_select_limit = 1000
+
+# Environment Variable
+GOOD_JOB_QUEUE_SELECT_LIMIT=1000
+```
+
+The Queue Select Limit value should be set to a rough upper-bound that exceeds all GoodJob execution threads / database connections. `1000` is a number that likely exceeds the available database connections on most PaaS offerings, but still offers a performance boost for GoodJob when executing very large queues.
+
+To explain where this value is used, here is the pseudo-query that GoodJob uses to find executable jobs:
+
+```sql
+  SELECT *
+  FROM good_jobs
+  WHERE id IN (
+    WITH rows AS MATERIALIZED (
+      SELECT id, active_job_id
+      FROM good_jobs
+      WHERE (scheduled_at <= NOW() OR scheduled_at IS NULL) AND finished_at IS NULL
+      ORDER BY priority DESC NULLS LAST, created_at ASC
+      [LIMIT 1000] -- <= introduced when queue_select_limit is set
+    )
+    SELECT id
+    FROM rows
+    WHERE pg_try_advisory_lock(('x' || substr(md5('good_jobs' || '-' || active_job_id::text), 1, 16))::bit(64)::bigint)
+    LIMIT 1
+  )
+```
+
 ### Execute jobs async / in-process
 
 GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:

data/app/models/good_job/execution.rb

@@ -66,6 +66,7 @@ module GoodJob
     end
 
     belongs_to :job, class_name: 'GoodJob::Job', foreign_key: 'active_job_id', primary_key: 'active_job_id', optional: true, inverse_of: :executions
+    after_destroy -> { self.class.active_job_id(active_job_id).delete_all }, if: -> { @_destroy_job }
 
     # Get executions with given ActiveJob ID
     # @!method active_job_id
@@ -203,10 +204,10 @@ module GoodJob
     #   return value for the job's +#perform+ method, and the exception the job
     #   raised, if any (if the job raised, then the second array entry will be
     #   +nil+). If there were no jobs to execute, returns +nil+.
-    def self.perform_with_advisory_lock(parsed_queues: nil)
+    def self.perform_with_advisory_lock(parsed_queues: nil, queue_select_limit: nil)
       execution = nil
       result = nil
-      unfinished.dequeueing_ordered(parsed_queues).only_scheduled.limit(1).with_advisory_lock(unlock_session: true) do |executions|
+      unfinished.dequeueing_ordered(parsed_queues).only_scheduled.limit(1).with_advisory_lock(unlock_session: true, select_limit: queue_select_limit) do |executions|
         execution = executions.first
         break if execution.blank?
         break :unlocked unless execution&.executable?
@@ -296,13 +297,14 @@ module GoodJob
       job_error = result.handled_error || result.unhandled_error
       self.error = [job_error.class, ERROR_MESSAGE_SEPARATOR, job_error.message].join if job_error
 
+      reenqueued = result.retried? || retried_good_job_id.present?
       if result.unhandled_error && GoodJob.retry_on_unhandled_error
         save!
-      elsif GoodJob.preserve_job_records == true || (result.unhandled_error && GoodJob.preserve_job_records == :on_unhandled_error)
+      elsif GoodJob.preserve_job_records == true || reenqueued || (result.unhandled_error && GoodJob.preserve_job_records == :on_unhandled_error)
         self.finished_at = Time.current
         save!
       else
-        destroy!
+        destroy_job
       end
 
       result
@@ -354,6 +356,14 @@ module GoodJob
       (finished_at || Time.zone.now) - performed_at if performed_at
     end
 
+    # Destroys this execution and all executions within the same job
+    def destroy_job
+      @_destroy_job = true
+      destroy!
+    ensure
+      @_destroy_job = false
+    end
+
     private
 
     def active_job_data
@@ -379,7 +389,7 @@ module GoodJob
           end
           handled_error ||= current_thread.error_on_retry || current_thread.error_on_discard
 
-          ExecutionResult.new(value: value, handled_error: handled_error)
+          ExecutionResult.new(value: value, handled_error: handled_error, retried: current_thread.error_on_retry.present?)
         rescue StandardError => e
           ExecutionResult.new(value: nil, unhandled_error: e)
         end

data/app/models/good_job/execution_result.rb

@@ -8,14 +8,18 @@ module GoodJob
     attr_reader :handled_error
     # @return [Exception, nil]
     attr_reader :unhandled_error
+    # @return [Exception, nil]
+    attr_reader :retried
+    alias retried? retried
 
     # @param value [Object, nil]
     # @param handled_error [Exception, nil]
     # @param unhandled_error [Exception, nil]
-    def initialize(value:, handled_error: nil, unhandled_error: nil)
+    def initialize(value:, handled_error: nil, unhandled_error: nil, retried: false)
       @value = value
       @handled_error = handled_error
       @unhandled_error = unhandled_error
+      @retried = retried
     end
   end
 end

data/app/models/good_job/lockable.rb

@@ -37,11 +37,12 @@ module GoodJob
       # @param function [String, Symbol]  Postgres Advisory Lock function name to use
       # @return [ActiveRecord::Relation]
       #   A relation selecting only the records that were locked.
-      scope :advisory_lock, (lambda do |column: _advisory_lockable_column, function: advisory_lockable_function|
+      scope :advisory_lock, (lambda do |column: _advisory_lockable_column, function: advisory_lockable_function, select_limit: nil|
         original_query = self
 
         cte_table = Arel::Table.new(:rows)
         cte_query = original_query.select(primary_key, column).except(:limit)
+        cte_query = cte_query.limit(select_limit) if select_limit
         cte_type = if supports_cte_materialization_specifiers?
                      'MATERIALIZED'
                    else
@@ -154,10 +155,10 @@ module GoodJob
       #   MyLockableRecord.order(created_at: :asc).limit(2).with_advisory_lock do |record|
       #     do_something_with record
       #   end
-      def with_advisory_lock(column: _advisory_lockable_column, function: advisory_lockable_function, unlock_session: false)
+      def with_advisory_lock(column: _advisory_lockable_column, function: advisory_lockable_function, unlock_session: false, select_limit: nil)
         raise ArgumentError, "Must provide a block" unless block_given?
 
-        records = advisory_lock(column: column, function: function).to_a
+        records = advisory_lock(column: column, function: function, select_limit: select_limit).to_a
 
         begin
           unscoped { yield(records) }

data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb

@@ -41,5 +41,7 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
     add_index :good_jobs, [:cron_key, :cron_at], name: :index_good_jobs_on_cron_key_and_cron_at, unique: true
     add_index :good_jobs, [:active_job_id], name: :index_good_jobs_on_active_job_id
     add_index :good_jobs, [:finished_at], where: "retried_good_job_id IS NULL AND finished_at IS NOT NULL", name: :index_good_jobs_jobs_on_finished_at
+    add_index :good_jobs, [:priority, :created_at], order: { priority: "DESC NULLS LAST", created_at: :asc },
+      where: "finished_at IS NULL", name: :index_good_jobs_jobs_on_priority_created_at_when_unfinished
   end
 end

data/lib/generators/good_job/templates/update/migrations/03_create_index_good_jobs_jobs_on_priority_created_at_when_unfinished.rb.erb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+class CreateIndexGoodJobsJobsOnPriorityCreatedAtWhenUnfinished < ActiveRecord::Migration<%= migration_version %>
+  disable_ddl_transaction!
+
+   def change
+    reversible do |dir|
+      dir.up do
+        # Ensure this incremental update migration is idempotent
+        # with monolithic install migration.
+        return if connection.index_name_exists?(:good_jobs, :index_good_jobs_jobs_on_priority_created_at_when_unfinished)
+      end
+    end
+
+    add_index :good_jobs, [:priority, :created_at], order: { priority: "DESC NULLS LAST", created_at: :asc },
+      where: "finished_at IS NULL", name: :index_good_jobs_jobs_on_priority_created_at_when_unfinished,
+      algorithm: :concurrently
+  end
+end

data/lib/good_job/cli.rb

@@ -81,7 +81,12 @@ module GoodJob
                   desc: "Path to write daemonized Process ID (env var: GOOD_JOB_PIDFILE, default: tmp/pids/good_job.pid)"
     method_option :probe_port,
                   type: :numeric,
+                  banner: 'PORT',
                   desc: "Port for http health check (env var: GOOD_JOB_PROBE_PORT, default: nil)"
+    method_option :queue_select_limit,
+                  type: :numeric,
+                  banner: 'COUNT',
+                  desc: "The number of queued jobs to select when polling for a job to run. (env var: GOOD_JOB_QUEUE_SELECT_LIMIT, default: nil)"
 
     def start
       set_up_application!

data/lib/good_job/configuration.rb

@@ -208,6 +208,19 @@ module GoodJob
       cron.map { |cron_key, params| GoodJob::CronEntry.new(params.merge(key: cron_key)) }
     end
 
+    # The number of queued jobs to select when polling for a job to run.
+    # This limit is intended to avoid locking a large number of rows when selecting eligible jobs
+    # from the queue. This value should be higher than the total number of threads across all good_job
+    # processes to ensure a thread can retrieve an eligible and unlocked job.
+    # @return [Integer, nil]
+    def queue_select_limit
+      (
+        options[:queue_select_limit] ||
+        rails_config[:queue_select_limit] ||
+        env['GOOD_JOB_QUEUE_SELECT_LIMIT']
+      )&.to_i
+    end
+
     # Whether to destroy discarded jobs when cleaning up preserved jobs.
     # This configuration is only used when {GoodJob.preserve_job_records} is +true+.
     # @return [Boolean]

data/lib/good_job/job_performer.rb

@@ -24,7 +24,7 @@ module GoodJob
     # Perform the next eligible job
     # @return [Object, nil] Returns job result or +nil+ if no job was found
     def next
-      job_query.perform_with_advisory_lock(parsed_queues: parsed_queues)
+      job_query.perform_with_advisory_lock(parsed_queues: parsed_queues, queue_select_limit: GoodJob.configuration.queue_select_limit)
     end
 
     # Tests whether this performer should be used in GoodJob's current state.

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '3.5.1'
+  VERSION = '3.6.0'
 end

data/lib/good_job.rb

@@ -57,7 +57,7 @@ module GoodJob
   #   By default, GoodJob deletes job records after the job is completed successfully.
   #   If you want to preserve jobs for latter inspection, set this to +true+.
   #   If you want to preserve only jobs that finished with error for latter inspection, set this to +:on_unhandled_error+.
-  #   @return [Boolean, nil]
+  #   @return [Boolean, Symbol, nil]
   mattr_accessor :preserve_job_records, default: true
 
   # @!attribute [rw] retry_on_unhandled_error

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 3.5.1
+  version: 3.6.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-10-20 00:00:00.000000000 Z
+date: 2022-10-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -402,6 +402,7 @@ files:
 - lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb
 - lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb.erb
 - lib/generators/good_job/templates/update/migrations/02_create_good_job_settings.rb.erb
+- lib/generators/good_job/templates/update/migrations/03_create_index_good_jobs_jobs_on_priority_created_at_when_unfinished.rb.erb
 - lib/generators/good_job/update_generator.rb
 - lib/good_job.rb
 - lib/good_job/active_job_extensions/concurrency.rb