good_job 3.20.0 → 3.21.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 62d33742278a734dd514fc3147abd1393ee62c3fd644c4c065b10c4fc6a24755
-  data.tar.gz: 83d56214ec454cbc3d910d4cbefb0413877ca3b18577168c0d9b9b456385a7a3
+  metadata.gz: 4f34be14a5677efd6647698f22bcf5189528ca574f34ee5361647f79a02cbfbd
+  data.tar.gz: 575a2d5b98556a810b50a31192b3dde8cfd3ab8e9cf09ef57c6efc245e4dc425
 SHA512:
-  metadata.gz: 90a3730d4989f26837b0ac9cd8f15ea3029f0c1c474159a4798a58d695153cb2067c336d382f971ba42a6faf371f677f7b295741c8ab1557add7c6b403a1fa94
-  data.tar.gz: 684e5d8473b4f460707394fd670067b0aaf7a2749ed525be2893967c0953ed1c5afe4f35e8acfce4acaa7f7fa39ae00d16394b38d186a11f7eb8812839d65ff3
+  metadata.gz: 235c2d8c5519f6af13b5e98586b0e32f1436cf969ce35d8549cecb4e6e9ad172a01a2914e8da1928b37f87b1c47a9317ec3a66dd92412903e5808f308eb67753
+  data.tar.gz: b0c1f3ee48467b924dcde22d0529a8728873babf0202c531c615b3c3e5a64a201f094d77b9414cf532cb90e1d8610ebf07176a3e7688ca6639bf71ad52be2592

data/CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## [v3.21.1](https://github.com/bensheldon/good_job/tree/v3.21.1) (2023-11-14)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.21.0...v3.21.1)
+
+**Fixed bugs:**
+
+- Explicitly require `active_job/arguments` in `GoodJob::BatchRecord` [\#1150](https://github.com/bensheldon/good_job/pull/1150) ([hidenba](https://github.com/hidenba))
+- Bug: Polling only activates single thread, should eagerly create additional threads when jobs exist [\#1148](https://github.com/bensheldon/good_job/pull/1148) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Error when executing ActiveJob::Batch.new in Rails 7.1.x [\#1149](https://github.com/bensheldon/good_job/issues/1149)
+- Show whether or not cron scheduler is enable in dashboard \(UI\) [\#1117](https://github.com/bensheldon/good_job/issues/1117)
+- ActiveRecord::ConnectionNotEstablished For rails multi DB [\#1103](https://github.com/bensheldon/good_job/issues/1103)
+- Rails API dies when using latest good\_job version [\#952](https://github.com/bensheldon/good_job/issues/952)
+- config.good\_job.preserve\_job\_records = false not working with CRON [\#927](https://github.com/bensheldon/good_job/issues/927)
+- Pundit::NotDefinedError \(unable to find policy `Admin::GoodJob::JobPolicy` for \) - version 2.13.0 and above [\#618](https://github.com/bensheldon/good_job/issues/618)
+- Running CLI under foreman doesn't display log output until exit [\#490](https://github.com/bensheldon/good_job/issues/490)
+
+**Merged pull requests:**
+
+- Always instantiate MultiScheduler; delegate Scheduler Metrics to JobPerformer [\#1147](https://github.com/bensheldon/good_job/pull/1147) ([bensheldon](https://github.com/bensheldon))
+- Clarify on concurrency uniqueness constraints [\#1144](https://github.com/bensheldon/good_job/pull/1144) ([Earlopain](https://github.com/Earlopain))
+
+## [v3.21.0](https://github.com/bensheldon/good_job/tree/v3.21.0) (2023-11-06)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.20.0...v3.21.0)
+
+**Implemented enhancements:**
+
+- Add "cron enabled" column to processes index page [\#1127](https://github.com/bensheldon/good_job/pull/1127) ([bforma](https://github.com/bforma))
+- Add `limit:` kwarg to `GoodJob.perform_inline` [\#1126](https://github.com/bensheldon/good_job/pull/1126) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Cron scheduler and multiple processes [\#1128](https://github.com/bensheldon/good_job/issues/1128)
+- `GoodJob.on_thread_error` not called in tests [\#1102](https://github.com/bensheldon/good_job/issues/1102)
+
+**Merged pull requests:**
+
+- Use a Concurrent::Event for CLI signal-trapping loop [\#1141](https://github.com/bensheldon/good_job/pull/1141) ([bensheldon](https://github.com/bensheldon))
+- Update README's optimize queue explanation [\#1138](https://github.com/bensheldon/good_job/pull/1138) ([maestromac](https://github.com/maestromac))
+- Update development dependencies and light Rubocop'ing [\#1136](https://github.com/bensheldon/good_job/pull/1136) ([bensheldon](https://github.com/bensheldon))
+- Move the Rails app harness from `spec/test_app` to `demo` [\#1135](https://github.com/bensheldon/good_job/pull/1135) ([bensheldon](https://github.com/bensheldon))
+- In test, shutdown schedulers/capsules before doing assertions because of race conditions; store CI logs for Dev Env tests [\#1129](https://github.com/bensheldon/good_job/pull/1129) ([bensheldon](https://github.com/bensheldon))
+- Use a constant to represent `None` for default/blank memoizable values [\#1125](https://github.com/bensheldon/good_job/pull/1125) ([bensheldon](https://github.com/bensheldon))
+
 ## [v3.20.0](https://github.com/bensheldon/good_job/tree/v3.20.0) (2023-10-23)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v3.19.4...v3.20.0)

data/README.md

@@ -470,7 +470,7 @@ class MyJob < ApplicationJob
     # Can be String or Lambda/Proc that is invoked in the context of the job.
     # Note: Arguments passed to #perform_later can be accessed through ActiveJob's `arguments` method
     # which is an array containing positional arguments and, optionally, a kwarg hash.
-    key: -> { "Unique-#{arguments.first}-#{arguments.last[:version]}" } #  MyJob.perform_later("Alice", version: 'v2') => "Unique-Alice-v2"
+    key: -> { "MyJob-#{arguments.first}-#{arguments.last[:version]}" } #  MyJob.perform_later("Alice", version: 'v2') => "MyJob-Alice-v2"
   )
 
   def perform(first_name)
@@ -483,7 +483,7 @@ When testing, the resulting concurrency key value can be inspected:
 
 ```ruby
 job = MyJob.perform_later("Alice")
-job.good_job_concurrency_key #=> "Unique-Alice"
+job.good_job_concurrency_key #=> "MyJob-Alice"
 ```
 
 #### How concurrency controls work
@@ -939,7 +939,7 @@ By default, GoodJob creates a single thread execution pool that will execute job
 
     - `transactional_messages:2`: execute jobs enqueued on `transactional_messages`, with up to 2 threads.
     - `batch_processing:1` execute jobs enqueued on `batch_processing`, with a single thread.
-    - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing`, with up to 2 threads.
+    - `-transactional_messages,batch_processing:2`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing`, with up to 2 threads.
     - `*`: execute jobs on any queue, with up to 5 threads (as configured by `--max-threads=5`).
 
     When a pool is performing jobs from multiple queues, jobs will be performed from specified queues, ordered by priority and creation time. To perform jobs from queues in the queues' given order, use the `+` modifier. In this example, jobs in `batch_processing` will be performed only when there are no jobs in `transactional_messages`:
@@ -1370,7 +1370,7 @@ bin/setup
 
 #### Rails development harness
 
-A Rails application exists within `spec/test_app` that is used for development, test, and GoodJob Demo environments.
+A Rails application exists within `demo` that is used for development, test, and GoodJob Demo environments.
 
 ```bash
 # Run a local development webserver

data/app/models/good_job/batch_record.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'active_job/arguments'
+
 module GoodJob
   class BatchRecord < BaseRecord
     include AdvisoryLockable

data/app/models/good_job/execution.rb

@@ -248,6 +248,7 @@ module GoodJob
 
     # Finds the next eligible Execution, acquire an advisory lock related to it, and
     # executes the job.
+    # @yield [Execution, nil] The next eligible Execution, or +nil+ if none found, before it is performed.
     # @return [ExecutionResult, nil]
     #   If a job was executed, returns an array with the {Execution} record, the
     #   return value for the job's +#perform+ method, and the exception the job
@@ -256,21 +257,19 @@ module GoodJob
     def self.perform_with_advisory_lock(parsed_queues: nil, queue_select_limit: nil)
       execution = nil
       result = nil
+
       unfinished.dequeueing_ordered(parsed_queues).only_scheduled.limit(1).with_advisory_lock(select_limit: queue_select_limit) do |executions|
         execution = executions.first
-        break if execution.blank?
-
-        unless execution.executable?
-          result = ExecutionResult.new(value: nil, unexecutable: true)
+        if execution&.executable?
+          yield(execution) if block_given?
+          result = execution.perform
+        else
           execution = nil
-          break
+          yield(nil) if block_given?
         end
-
-        yield(execution) if block_given?
-        result = execution.perform
       end
-      execution&.run_callbacks(:perform_unlocked)
 
+      execution&.run_callbacks(:perform_unlocked)
       result
     end
 

data/app/models/good_job/process.rb

@@ -52,6 +52,10 @@ module GoodJob # :nodoc:
     end
 
     def self.ns_current_state
+      total_succeeded_executions_count = GoodJob::Scheduler.instances.sum { |scheduler| scheduler.stats.fetch(:succeeded_executions_count, 0) }
+      total_errored_executions_count = GoodJob::Scheduler.instances.sum { |scheduler| scheduler.stats.fetch(:errored_executions_count, 0) }
+      total_empty_executions_count = GoodJob::Scheduler.instances.sum { |scheduler| scheduler.stats.fetch(:empty_executions_count, 0) }
+
       {
         id: ns_current_id,
         hostname: Socket.gethostname,
@@ -61,8 +65,10 @@ module GoodJob # :nodoc:
         retry_on_unhandled_error: GoodJob.retry_on_unhandled_error,
         schedulers: GoodJob::Scheduler.instances.map(&:stats),
         cron_enabled: GoodJob.configuration.enable_cron?,
-        total_succeeded_executions_count: GoodJob::Scheduler.instances.sum { |scheduler| scheduler.stats.fetch(:succeeded_executions_count) },
-        total_errored_executions_count: GoodJob::Scheduler.instances.sum { |scheduler| scheduler.stats.fetch(:errored_executions_count) },
+        total_succeeded_executions_count: total_succeeded_executions_count,
+        total_errored_executions_count: total_errored_executions_count,
+        total_executions_count: total_succeeded_executions_count + total_errored_executions_count,
+        total_empty_executions_count: total_empty_executions_count,
         database_connection_pool: {
           size: connection_pool.size,
           active: connection_pool.connections.count(&:in_use?),

data/app/views/good_job/processes/index.html.erb

@@ -8,6 +8,7 @@
       <div class="row small text-muted text-uppercase align-items-center">
         <div class="col"><%= t ".process" %></div>
         <div class="col"><%= t ".schedulers" %></div>
+        <div class="col-2 d-flex gap-2"><%= t ".cron_enabled" %></div>
         <div class="col-2 d-flex gap-2">
           <span><%= t ".started" %></span>
         </div>
@@ -44,6 +45,7 @@
               <pre class="mb-0"><%= scheduler.is_a?(Hash) ? scheduler['name'] : scheduler %></pre>
             <% end %>
           </div>
+          <div class="col-2 small"><%= t(ActiveModel::Type::Boolean.new.cast(process.state["cron_enabled"]), scope: "good_job.shared.boolean") %></div>
           <div class="col-2 small"><%= relative_time(process.created_at) %></div>
           <div class="col-2 small"><%= relative_time(process.updated_at) %></div>
           <div class="col-auto">

data/config/locales/de.yml

@@ -198,6 +198,7 @@ de:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron aktiviert
         no_good_job_processes_found: Keine GoodJob-Prozesse gefunden.
         process: Verfahren
         schedulers: Planer
@@ -205,6 +206,9 @@ de:
         title: Prozesse
         updated: Aktualisiert
     shared:
+      boolean:
+        'false': Nein
+        'true': Ja
       error: Fehler
       filter:
         all: Alle

data/config/locales/en.yml

@@ -198,6 +198,7 @@ en:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron enabled
         no_good_job_processes_found: No GoodJob processes found.
         process: Process
         schedulers: Schedulers
@@ -205,6 +206,9 @@ en:
         title: Processes
         updated: Updated
     shared:
+      boolean:
+        'false': 'No'
+        'true': 'Yes'
       error: Error
       filter:
         all: All

data/config/locales/es.yml

@@ -196,6 +196,7 @@ es:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron habilitado
         no_good_job_processes_found: No hay procesos de GoodJob.
         process: Proceso
         schedulers: Schedulers
@@ -203,6 +204,9 @@ es:
         title: Procesos
         updated: Actualizado
     shared:
+      boolean:
+        'false': 'No'
+        'true': Sí
       error: Error
       filter:
         all: Todas

data/config/locales/fr.yml

@@ -198,6 +198,7 @@ fr:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron activé
         no_good_job_processes_found: Aucun processus GoodJob trouvé.
         process: Processus
         schedulers: Schedulers
@@ -205,6 +206,9 @@ fr:
         title: Processus
         updated: Mis à jour
     shared:
+      boolean:
+        'false': Non
+        'true': Oui
       error: Erreur
       filter:
         all: Tous

data/config/locales/ja.yml

@@ -198,6 +198,7 @@ ja:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron が有効になっている
         no_good_job_processes_found: GoodJobのプロセスが見つかりませんでした。
         process: プロセス
         schedulers: スケジューラー
@@ -205,6 +206,9 @@ ja:
         title: プロセス
         updated: 更新された
     shared:
+      boolean:
+        'false': いいえ
+        'true': はい
       error: エラー
       filter:
         all: 全て

data/config/locales/nl.yml

@@ -198,6 +198,7 @@ nl:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron ingeschakeld
         no_good_job_processes_found: Geen GoodJob-processen gevonden.
         process: Proces
         schedulers: Planners
@@ -205,6 +206,9 @@ nl:
         title: Processen
         updated: Bijgewerkt
     shared:
+      boolean:
+        'false': Nee
+        'true': Ja
       error: Fout
       filter:
         all: Alle

data/config/locales/ru.yml

@@ -224,6 +224,7 @@ ru:
             unit: ''
     processes:
       index:
+        cron_enabled: Крон включен
         no_good_job_processes_found: Процессы GoodJob не найдены.
         process: Процесс
         schedulers: Планировщики
@@ -231,6 +232,9 @@ ru:
         title: Процессы
         updated: Обновлено
     shared:
+      boolean:
+        'false': Нет
+        'true': Да
       error: Ошибка
       filter:
         all: Все

data/config/locales/tr.yml

@@ -198,6 +198,7 @@ tr:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron etkin
         no_good_job_processes_found: GoodJob süreci bulunamadı.
         process: Süreç
         schedulers: Planlayıcılar
@@ -205,6 +206,9 @@ tr:
         title: Süreçler
         updated: Güncellenmiş
     shared:
+      boolean:
+        'false': Hayır
+        'true': Evet
       error: Hata
       filter:
         all: Tümü

data/config/locales/uk.yml

@@ -224,6 +224,7 @@ uk:
             unit: ''
     processes:
       index:
+        cron_enabled: Cron увімкнено
         no_good_job_processes_found: Процеси GoodJob не знайдені.
         process: Процес
         schedulers: Планувальники
@@ -231,6 +232,9 @@ uk:
         title: Процеси
         updated: Оновлено
     shared:
+      boolean:
+        'false': Ні
+        'true': Так
       error: Помилка
       filter:
         all: Всі

data/lib/good_job/adapter.rb

@@ -168,13 +168,13 @@ module GoodJob
     end
 
     # Shut down the thread pool executors.
-    # @param timeout [nil, Numeric, Symbol] Seconds to wait for active threads.
+    # @param timeout [nil, Numeric, NONE] Seconds to wait for active threads.
     #   * +nil+ trigger a shutdown but not wait for it to complete.
     #   * +-1+ wait until the shutdown is complete.
     #   * +0+ immediately shutdown and stop any threads.
     #   * A positive number will wait that many seconds before stopping any remaining active threads.
     # @return [void]
-    def shutdown(timeout: :default)
+    def shutdown(timeout: NONE)
       @capsule&.shutdown(timeout: timeout)
       @_async_started = false
     end

data/lib/good_job/capsule.rb

@@ -32,9 +32,9 @@ module GoodJob
         @shared_executor = GoodJob::SharedExecutor.new
         @notifier = GoodJob::Notifier.new(enable_listening: @configuration.enable_listen_notify, executor: @shared_executor.executor)
         @poller = GoodJob::Poller.new(poll_interval: @configuration.poll_interval)
-        @scheduler = GoodJob::Scheduler.from_configuration(@configuration, warm_cache_on_initialize: true)
-        @notifier.recipients << [@scheduler, :create_thread]
-        @poller.recipients << [@scheduler, :create_thread]
+        @multi_scheduler = GoodJob::MultiScheduler.from_configuration(@configuration, warm_cache_on_initialize: true)
+        @notifier.recipients.push([@multi_scheduler, :create_thread])
+        @poller.recipients.push(-> { @multi_scheduler.create_thread({ fanout: true }) })
 
         @cron_manager = GoodJob::CronManager.new(@configuration.cron_entries, start_on_initialize: true, executor: @shared_executor.executor) if @configuration.enable_cron?
 
@@ -44,23 +44,23 @@ module GoodJob
     end
 
     # Shut down the thread pool executors.
-    # @param timeout [nil, Numeric, Symbol] Seconds to wait for active threads.
+    # @param timeout [nil, Numeric, NONE] Seconds to wait for active threads.
     #   * +-1+ will wait for all active threads to complete.
     #   * +0+ will interrupt active threads.
     #   * +N+ will wait at most N seconds and then interrupt active threads.
     #   * +nil+ will trigger a shutdown but not wait for it to complete.
     # @return [void]
-    def shutdown(timeout: :default)
-      timeout = @configuration.shutdown_timeout if timeout == :default
-      GoodJob._shutdown_all([@shared_executor, @notifier, @poller, @scheduler, @cron_manager].compact, timeout: timeout)
+    def shutdown(timeout: NONE)
+      timeout = @configuration.shutdown_timeout if timeout == NONE
+      GoodJob._shutdown_all([@shared_executor, @notifier, @poller, @multi_scheduler, @cron_manager].compact, timeout: timeout)
       @startable = false
       @running = false
     end
 
     # Shutdown and then start the capsule again.
-    # @param timeout [Numeric, Symbol] Seconds to wait for active threads.
+    # @param timeout [Numeric, NONE] Seconds to wait for active threads.
     # @return [void]
-    def restart(timeout: :default)
+    def restart(timeout: NONE)
       raise ArgumentError, "Capsule#restart cannot be called with a timeout of nil" if timeout.nil?
 
       shutdown(timeout: timeout)
@@ -74,7 +74,7 @@ module GoodJob
 
     # @return [Boolean] Whether the capsule has been shutdown.
     def shutdown?
-      [@shared_executor, @notifier, @poller, @scheduler, @cron_manager].compact.all?(&:shutdown?)
+      [@shared_executor, @notifier, @poller, @multi_scheduler, @cron_manager].compact.all?(&:shutdown?)
     end
 
     # Creates an execution thread(s) with the given attributes.
@@ -82,7 +82,7 @@ module GoodJob
     # @return [Boolean, nil] Whether the thread was created.
     def create_thread(job_state = nil)
       start if startable?
-      @scheduler&.create_thread(job_state)
+      @multi_scheduler&.create_thread(job_state)
     end
 
     private

data/lib/good_job/cli.rb

@@ -17,6 +17,9 @@ module GoodJob
     # Requiring this loads the application's configuration and classes.
     RAILS_ENVIRONMENT_RB = File.expand_path("config/environment.rb")
 
+    # Number of seconds between checking shutdown conditions
+    SHUTDOWN_EVENT_TIMEOUT = 10
+
     class << self
       # Whether the CLI is running from the executable
       # @return [Boolean, nil]
@@ -106,14 +109,15 @@ module GoodJob
         probe_server.start
       end
 
-      @stop_good_job_executable = false
+      require 'concurrent/atomic/event'
+      @stop_good_job_executable = Concurrent::Event.new
       %w[INT TERM].each do |signal|
-        trap(signal) { @stop_good_job_executable = true }
+        trap(signal) { Thread.new { @stop_good_job_executable.set }.join }
       end
 
       Kernel.loop do
-        sleep 0.1
-        break if @stop_good_job_executable || capsule.shutdown?
+        @stop_good_job_executable.wait(SHUTDOWN_EVENT_TIMEOUT)
+        break if @stop_good_job_executable.set? || capsule.shutdown?
       end
 
       systemd.stop do

data/lib/good_job/job_performer/metrics.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require 'concurrent/atomic/atomic_fixnum'
+
+module GoodJob # :nodoc:
+  class JobPerformer
+    # Metrics for the scheduler.
+    class Metrics
+      def initialize
+        @mutex = Mutex.new
+        @empty_executions = Concurrent::AtomicFixnum.new
+        @errored_executions = Concurrent::AtomicFixnum.new
+        @succeeded_executions = Concurrent::AtomicFixnum.new
+        @execution_at = nil
+        @check_queue_at = nil
+      end
+
+      # Increments number of failed executions.
+      # @return [Integer]
+      def increment_errored_executions
+        @execution_at = Time.current
+        @errored_executions.increment
+      end
+
+      # Increments number of succeeded executions.
+      # @return [Integer]
+      def increment_succeeded_executions
+        @execution_at = Time.current
+        @succeeded_executions.increment
+      end
+
+      # Increments number of dequeue attempts with no executions.
+      # @return [Integer]
+      def increment_empty_executions
+        @execution_at = Time.current
+        @empty_executions.increment
+      end
+
+      # Last time a job was executed (started or finished).
+      # @return [Time, nil]
+      def touch_execution_at
+        @execution_at = Time.current
+      end
+
+      # Last time the queue was checked for jobs.
+      # @return [Time, nil]
+      def touch_check_queue_at
+        @check_queue_at = Time.current
+      end
+
+      # All metrics in a Hash.
+      # @return [Hash]
+      def to_h
+        {
+          empty_executions_count: @empty_executions.value,
+          errored_executions_count: @errored_executions.value,
+          succeeded_executions_count: @succeeded_executions.value,
+        }.tap do |values|
+          values[:total_executions_count] = values[:succeeded_executions_count] + values[:errored_executions_count]
+          values[:execution_at] = @execution_at
+          values[:check_queue_at] = @check_queue_at
+        end
+      end
+
+      # Reset counters.
+      # @return [void]
+      def reset
+        @empty_executions.value = 0
+        @errored_executions.value = 0
+        @succeeded_executions.value = 0
+        @execution_at = nil
+        @check_queue_at = nil
+      end
+    end
+  end
+end

data/lib/good_job/job_performer.rb

@@ -16,6 +16,7 @@ module GoodJob
     # @param queue_string [String] Queues to execute jobs from
     def initialize(queue_string)
       @queue_string = queue_string
+      @metrics = Metrics.new
     end
 
     # A meaningful name to identify the performer in logs and for debugging.
@@ -25,15 +26,28 @@ module GoodJob
     end
 
     # Perform the next eligible job
+    # @yield [Execution] Yields the execution, if one is dequeued
     # @return [Object, nil] Returns job result or +nil+ if no job was found
     def next
       active_job_id = nil
       job_query.perform_with_advisory_lock(parsed_queues: parsed_queues, queue_select_limit: GoodJob.configuration.queue_select_limit) do |execution|
-        active_job_id = execution.active_job_id
-        performing_active_job_ids << active_job_id
+        @metrics.touch_check_queue_at
+
+        if execution
+          active_job_id = execution.active_job_id
+          performing_active_job_ids << active_job_id
+          @metrics.touch_execution_at
+          yield(execution) if block_given?
+        else
+          @metrics.increment_empty_executions
+        end
+      end.tap do |result|
+        if result
+          result.succeeded? ? @metrics.increment_succeeded_executions : @metrics.increment_errored_executions
+        end
       end
     ensure
-      performing_active_job_ids.delete(active_job_id)
+      performing_active_job_ids.delete(active_job_id) if active_job_id
     end
 
     # Tests whether this performer should be used in GoodJob's current state.
@@ -72,6 +86,20 @@ module GoodJob
       GoodJob.cleanup_preserved_jobs
     end
 
+    # Metrics about this performer
+    # @return [Hash]
+    def stats
+      {
+        name: name,
+      }.merge(@metrics.to_h)
+    end
+
+    # Reset metrics about this performer
+    # @return [void]
+    def reset_stats
+      @metrics.reset
+    end
+
     private
 
     attr_reader :queue_string

data/lib/good_job/multi_scheduler.rb

@@ -3,6 +3,29 @@
 module GoodJob
   # Delegates the interface of a single {Scheduler} to multiple Schedulers.
   class MultiScheduler
+    # Creates MultiScheduler from a GoodJob::Configuration instance.
+    # @param configuration [GoodJob::Configuration]
+    # @param warm_cache_on_initialize [Boolean]
+    # @return [GoodJob::MultiScheduler]
+    def self.from_configuration(configuration, warm_cache_on_initialize: false)
+      schedulers = configuration.queue_string.split(';').map do |queue_string_and_max_threads|
+        queue_string, max_threads = queue_string_and_max_threads.split(':')
+        max_threads = (max_threads || configuration.max_threads).to_i
+
+        job_performer = GoodJob::JobPerformer.new(queue_string)
+        GoodJob::Scheduler.new(
+          job_performer,
+          max_threads: max_threads,
+          max_cache: configuration.max_cache,
+          warm_cache_on_initialize: warm_cache_on_initialize,
+          cleanup_interval_seconds: configuration.cleanup_interval_seconds,
+          cleanup_interval_jobs: configuration.cleanup_interval_jobs
+        )
+      end
+
+      new(schedulers)
+    end
+
     # @return [Array<Scheduler>] List of the scheduler delegates
     attr_reader :schedulers
 
@@ -43,7 +66,7 @@ module GoodJob
     def create_thread(state = nil)
       results = []
 
-      if state
+      if state && !state[:fanout]
         schedulers.any? do |scheduler|
           scheduler.create_thread(state).tap { |result| results << result }
         end
@@ -61,5 +84,19 @@ module GoodJob
         nil
       end
     end
+
+    def stats
+      scheduler_stats = schedulers.map(&:stats)
+
+      {
+        schedulers: scheduler_stats,
+        empty_executions_count: scheduler_stats.sum { |stats| stats.fetch(:empty_executions_count, 0) },
+        errored_executions_count: scheduler_stats.sum { |stats| stats.fetch(:errored_executions_count, 0) },
+        succeeded_executions_count: scheduler_stats.sum { |stats| stats.fetch(:succeeded_executions_count, 0) },
+        total_executions_count: scheduler_stats.sum { |stats| stats.fetch(:total_executions_count, 0) },
+        execution_at: scheduler_stats.map { |stats| stats.fetch(:execution_at, nil) }.compact.max,
+        check_queue_at: scheduler_stats.map { |stats| stats.fetch(:check_queue_at, nil) }.compact.max,
+      }
+    end
   end
 end

data/lib/good_job/scheduler.rb

@@ -4,7 +4,6 @@ require "concurrent/executor/thread_pool_executor"
 require "concurrent/executor/timer_set"
 require "concurrent/scheduled_task"
 require "concurrent/utility/processor_counter"
-require 'good_job/metrics'
 
 module GoodJob # :nodoc:
   #
@@ -36,33 +35,6 @@ module GoodJob # :nodoc:
     #   @return [Array<GoodJob::Scheduler>, nil]
     cattr_reader :instances, default: Concurrent::Array.new, instance_reader: false
 
-    # Creates GoodJob::Scheduler(s) and Performers from a GoodJob::Configuration instance.
-    # @param configuration [GoodJob::Configuration]
-    # @param warm_cache_on_initialize [Boolean]
-    # @return [GoodJob::Scheduler, GoodJob::MultiScheduler]
-    def self.from_configuration(configuration, warm_cache_on_initialize: false)
-      schedulers = configuration.queue_string.split(';').map do |queue_string_and_max_threads|
-        queue_string, max_threads = queue_string_and_max_threads.split(':')
-        max_threads = (max_threads || configuration.max_threads).to_i
-
-        job_performer = GoodJob::JobPerformer.new(queue_string)
-        GoodJob::Scheduler.new(
-          job_performer,
-          max_threads: max_threads,
-          max_cache: configuration.max_cache,
-          warm_cache_on_initialize: warm_cache_on_initialize,
-          cleanup_interval_seconds: configuration.cleanup_interval_seconds,
-          cleanup_interval_jobs: configuration.cleanup_interval_jobs
-        )
-      end
-
-      if schedulers.size > 1
-        GoodJob::MultiScheduler.new(schedulers)
-      else
-        schedulers.first
-      end
-    end
-
     # Human readable name of the scheduler that includes configuration values.
     # @return [String]
     attr_reader :name
@@ -88,7 +60,6 @@ module GoodJob # :nodoc:
       @executor_options[:name] = name
 
       @cleanup_tracker = CleanupTracker.new(cleanup_interval_seconds: cleanup_interval_seconds, cleanup_interval_jobs: cleanup_interval_jobs)
-      @metrics = ::GoodJob::Metrics.new
       @executor_options[:name] = name
 
       create_executor
@@ -143,7 +114,7 @@ module GoodJob # :nodoc:
 
       instrument("scheduler_restart_pools") do
         shutdown(timeout: timeout)
-        @metrics.reset
+        @performer.reset_stats
         create_executor
         warm_cache
       end
@@ -152,16 +123,17 @@ module GoodJob # :nodoc:
     # Wakes a thread to allow the performer to execute a task.
     # @param state [Hash, nil] Contextual information for the performer. See {JobPerformer#next?}.
     # @return [Boolean, nil] Whether work was started.
-    #
     #   * +nil+ if the scheduler is unable to take new work, for example if the thread pool is shut down or at capacity.
     #   * +true+ if the performer started executing work.
     #   * +false+ if the performer decides not to attempt to execute a task based on the +state+ that is passed to it.
     def create_thread(state = nil)
       return nil unless executor.running?
 
-      if state
+      if state.present?
         return false unless performer.next?(state)
 
+        fanout = state&.fetch(:fanout, nil)
+
         if state[:count]
           # When given state for multiple jobs, try to create a thread for each one.
           # Return true if a thread can be created for all of them, nil if partial or none.
@@ -193,7 +165,7 @@ module GoodJob # :nodoc:
         return nil unless remaining_cache_count.positive?
       end
 
-      create_task(delay)
+      create_task(delay, fanout: fanout)
 
       run_now ? true : nil
     end
@@ -207,16 +179,6 @@ module GoodJob # :nodoc:
       unhandled_error = thread_error || result&.unhandled_error
       GoodJob._on_thread_error(unhandled_error) if unhandled_error
 
-      if unhandled_error || result&.handled_error
-        @metrics.increment_errored_executions
-      elsif result&.unexecutable
-        @metrics.increment_unexecutable_executions
-      elsif result
-        @metrics.increment_succeeded_executions
-      else
-        @metrics.increment_empty_executions
-      end
-
       instrument("finished_job_task", { result: output, error: thread_error, time: time })
       return unless output
 
@@ -232,6 +194,7 @@ module GoodJob # :nodoc:
     # @return [Hash]
     def stats
       available_threads = executor.ready_worker_count
+
       {
         name: name,
         queues: performer.name,
@@ -241,7 +204,7 @@ module GoodJob # :nodoc:
         max_cache: @max_cache,
         active_cache: cache_count,
         available_cache: remaining_cache_count,
-      }.merge!(@metrics.to_h)
+      }.merge!(@performer.stats.without(:name))
     end
 
     # Preload existing runnable and future-scheduled jobs
@@ -300,12 +263,15 @@ module GoodJob # :nodoc:
     end
 
     # @param delay [Integer]
+    # @param fanout [Boolean] Whether to eagerly create a 2nd execution thread if a job is found.
     # @return [void]
-    def create_task(delay = 0)
-      future = Concurrent::ScheduledTask.new(delay, args: [performer], executor: executor, timer_set: timer_set) do |thr_performer|
+    def create_task(delay = 0, fanout: false)
+      future = Concurrent::ScheduledTask.new(delay, args: [self, performer], executor: executor, timer_set: timer_set) do |thr_scheduler, thr_performer|
         Thread.current.name = Thread.current.name.sub("-worker-", "-thread-") if Thread.current.name
         Rails.application.reloader.wrap do
-          thr_performer.next
+          thr_performer.next do |found|
+            thr_scheduler.create_thread({ fanout: fanout }) if found && fanout
+          end
         end
       end
       future.add_observer(self, :task_observer)

data/lib/good_job/version.rb

@@ -2,7 +2,7 @@
 
 module GoodJob
   # GoodJob gem version.
-  VERSION = '3.20.0'
+  VERSION = '3.21.1'
 
   # GoodJob version as Gem::Version object
   GEM_VERSION = Gem::Version.new(VERSION)

data/lib/good_job.rb

@@ -26,6 +26,7 @@ require 'good_job/current_thread'
 require "good_job/daemon"
 require "good_job/dependencies"
 require "good_job/job_performer"
+require "good_job/job_performer/metrics"
 require "good_job/log_subscriber"
 require "good_job/multi_scheduler"
 require "good_job/notifier"
@@ -42,6 +43,10 @@ require "good_job/systemd_service"
 module GoodJob
   include GoodJob::Dependencies
 
+  # Default, null, blank value placeholder.
+  NONE = Module.new.freeze
+
+  # Default logger for GoodJob; overridden by Rails.logger in Railtie.
   DEFAULT_LOGGER = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new($stdout))
 
   # @!attribute [rw] active_record_parent_class
@@ -233,13 +238,19 @@ module GoodJob
   # This is primarily intended for usage in a test environment.
   # Unhandled job errors will be raised.
   # @param queue_string [String] Queues to execute jobs from
+  # @param limit [Integer, nil] Maximum number of iterations for the loop
   # @return [void]
-  def self.perform_inline(queue_string = "*")
+  def self.perform_inline(queue_string = "*", limit: nil)
     job_performer = JobPerformer.new(queue_string)
+    iteration = 0
     loop do
+      break if limit && iteration >= limit
+
       result = Rails.application.reloader.wrap { job_performer.next }
       break unless result
       raise result.unhandled_error if result.unhandled_error
+
+      iteration += 1
     end
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 3.20.0
+  version: 3.21.1
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-10-23 00:00:00.000000000 Z
+date: 2023-11-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -366,8 +366,8 @@ files:
 - lib/good_job/http_server.rb
 - lib/good_job/interrupt_error.rb
 - lib/good_job/job_performer.rb
+- lib/good_job/job_performer/metrics.rb
 - lib/good_job/log_subscriber.rb
-- lib/good_job/metrics.rb
 - lib/good_job/multi_scheduler.rb
 - lib/good_job/notifier.rb
 - lib/good_job/notifier/process_heartbeat.rb

data/lib/good_job/metrics.rb

@@ -1,57 +0,0 @@
-# frozen_string_literal: true
-
-module GoodJob # :nodoc:
-  # Metrics for the scheduler.
-  class Metrics
-    def initialize
-      @empty_executions = Concurrent::AtomicFixnum.new
-      @errored_executions = Concurrent::AtomicFixnum.new
-      @succeeded_executions = Concurrent::AtomicFixnum.new
-      @unexecutable_executions = Concurrent::AtomicFixnum.new
-    end
-
-    # Increments number of empty queried executions.
-    # @return [Integer]
-    def increment_empty_executions
-      @empty_executions.increment
-    end
-
-    # Increments number of failed executions.
-    # @return [Integer]
-    def increment_errored_executions
-      @errored_executions.increment
-    end
-
-    # Increments number of succeeded executions.
-    # @return [Integer]
-    def increment_succeeded_executions
-      @succeeded_executions.increment
-    end
-
-    # Increments number of unlocked executions.
-    # @return [Integer]
-    def increment_unexecutable_executions
-      @unexecutable_executions.increment
-    end
-
-    def to_h
-      {
-        empty_executions_count: @empty_executions.value,
-        errored_executions_count: @errored_executions.value,
-        succeeded_executions_count: @succeeded_executions.value,
-        unexecutable_executions_count: @unexecutable_executions.value,
-      }.tap do |values|
-        values[:total_executions_count] = values.values.sum
-      end
-    end
-
-    # Reset counters.
-    # @return [void]
-    def reset
-      @empty_executions.value = 0
-      @errored_executions.value = 0
-      @succeeded_executions.value = 0
-      @unexecutable_executions.value = 0
-    end
-  end
-end