good_job 3.4.0 → 3.4.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 73efcb35d2a5e0ccfd13e2526a9a9d4a7195c4bf2e4a7f4127894a68c7f5dbb9
-  data.tar.gz: d9fddab346e5851e23daa5d9bef8c68b1fd5472f37d687c3cb49c5ff551cc31c
+  metadata.gz: 8838f9cf09b7dda04c3dc119db41deef4facc8b31a07e879e082898d77915d8d
+  data.tar.gz: a7e146e678ea6432f291f593edfce11497b70201576e299360a026dc83290dda
 SHA512:
-  metadata.gz: d720ab703536db9dfd396097aeb8b3aae64f5cd55a1754a9965eaaca275a2babfc4a05d249fb13896d776d02034553bd81cdda8b0af19a6f49d9a3a4bca29045
-  data.tar.gz: 9dd2af1f00637c3db171934777cc02be231e3ee98ad953aa9fcf030b00faa9fdd2b811f6e69094120a2deee8415e6a33d2d2a4a4d7701fbf89f7c45e272658ef
+  metadata.gz: 87a1770d3b74c50c6a40bc63ca50ad368578524029b8d11b5221baf721f7c8169051f0c19bc9554962ff61fc9cf51ac397f042ad393a4c19cecb73924db0c6c2
+  data.tar.gz: d707c9b35f41b417628425c74e916c21654992dcb41a27e52fdc88435e74fae09168b85bea81470aeba7eb2236b9d2d57a736ce4c34a5c427c4f82230146d7fa

data/CHANGELOG.md

@@ -1,5 +1,54 @@
 # Changelog
 
+## [v3.4.3](https://github.com/bensheldon/good_job/tree/v3.4.3) (2022-08-15)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.4.2...v3.4.3)
+
+**Closed issues:**
+
+- How to run multiple workers? [\#699](https://github.com/bensheldon/good_job/issues/699)
+- Getting Postgres Errors on killing development server after setting up Goodjob [\#692](https://github.com/bensheldon/good_job/issues/692)
+
+**Merged pull requests:**
+
+- Fix Project v2 GitHub Actions [\#701](https://github.com/bensheldon/good_job/pull/701) ([bensheldon](https://github.com/bensheldon))
+- Remove development dependencies: memory\_profiler, rbtrace, sigdump [\#700](https://github.com/bensheldon/good_job/pull/700) ([bensheldon](https://github.com/bensheldon))
+- Allow concurrency limits to be configured dynamically with lambda/proc [\#696](https://github.com/bensheldon/good_job/pull/696) ([baka-san](https://github.com/baka-san))
+- Add additional details to Concurrency Control explanation [\#695](https://github.com/bensheldon/good_job/pull/695) ([bensheldon](https://github.com/bensheldon))
+
+## [v3.4.2](https://github.com/bensheldon/good_job/tree/v3.4.2) (2022-08-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.4.1...v3.4.2)
+
+**Fixed bugs:**
+
+- Jobs enqueued via dashboard ignores app default\_locale [\#697](https://github.com/bensheldon/good_job/issues/697)
+- Include better exception log messages, including class and backtrace [\#693](https://github.com/bensheldon/good_job/pull/693) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Do we need to implement concurrency with scheduled cron jobs? [\#690](https://github.com/bensheldon/good_job/issues/690)
+- Uninitialized constant GoodJob::JobsController [\#674](https://github.com/bensheldon/good_job/issues/674)
+- ActiveRecord::StatementInvalid: PG::ConnectionBad: PQsocket\(\) can't get socket descriptor every 30 minutes aprox. [\#579](https://github.com/bensheldon/good_job/issues/579)
+- Handle assets in dashboard when rails app is behind proxy path [\#424](https://github.com/bensheldon/good_job/issues/424)
+
+**Merged pull requests:**
+
+- Enqueues Cron jobs with I18n default locale [\#698](https://github.com/bensheldon/good_job/pull/698) ([esasse](https://github.com/esasse))
+
+## [v3.4.1](https://github.com/bensheldon/good_job/tree/v3.4.1) (2022-08-06)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.4.0...v3.4.1)
+
+**Closed issues:**
+
+- Add `cron_enabled` to Process state [\#673](https://github.com/bensheldon/good_job/issues/673)
+- Good job is using a lot of memory / ram [\#613](https://github.com/bensheldon/good_job/issues/613)
+
+**Merged pull requests:**
+
+- Only report Notifier connection errors once after they happen 3 consecutive times [\#689](https://github.com/bensheldon/good_job/pull/689) ([bensheldon](https://github.com/bensheldon))
+
 ## [v3.4.0](https://github.com/bensheldon/good_job/tree/v3.4.0) (2022-08-05)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v3.3.3...v3.4.0)

data/README.md

@@ -406,15 +406,18 @@ class MyJob < ApplicationJob
 
   good_job_control_concurrency_with(
     # Maximum number of unfinished jobs to allow with the concurrency key
+    # Can be an Integer or Lambda/Proc that is invoked in the context of the job
     total_limit: 1,
 
     # Or, if more control is needed:
     # Maximum number of jobs with the concurrency key to be
     # concurrently enqueued (excludes performing jobs)
+    # Can be an Integer or Lambda/Proc that is invoked in the context of the job
     enqueue_limit: 2,
 
     # Maximum number of jobs with the concurrency key to be
     # concurrently performed (excludes enqueued jobs)
+    # Can be an Integer or Lambda/Proc that is invoked in the context of the job
     perform_limit: 1,
 
     # Note: Under heavy load, the total number of jobs may exceed the
@@ -444,10 +447,11 @@ job.good_job_concurrency_key #=> "Unique-Alice"
 
 #### How concurrency controls work
 
-GoodJob's concurrency control strategy for `perform_limit` is "optimistic retry with an incremental backoff". The [code is readable](https://github.com/bensheldon/good_job/blob/main/lib/good_job/active_job_extensions/concurrency.rb).
+GoodJob's concurrency control strategy for `perform_limit` is "optimistic retry with an incremental backoff".  The [code is readable](https://github.com/bensheldon/good_job/blob/main/lib/good_job/active_job_extensions/concurrency.rb).
 
-- "Optimistic" meaning that the implementation's performance trade-off assumes that collisions are atypical (e.g. two users enqueue the same job at the same time) rather than regular (e.g. the system enqueues thousands of colliding jobs at the same time).
+- "Optimistic" meaning that the implementation's performance trade-off assumes that collisions are atypical (e.g. two users enqueue the same job at the same time) rather than regular (e.g. the system enqueues thousands of colliding jobs at the same time). Depending on your concurrency requirements, you may also want to manage concurrency through the number of GoodJob threads and processes that are performing a given queue.
 - "Retry with an incremental backoff" means that when `perform_limit` is exceeded, the job will raise a `GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError` which is caught by a `retry_on` handler which re-schedules the job to execute in the near future with an incremental backoff.
+- First-in-first-out job execution order is not preserved when a job is retried with incremental back-off.
 
 ### Cron-style repeating/recurring jobs
 

data/app/models/good_job/cron_entry.rb

@@ -106,7 +106,9 @@ module GoodJob # :nodoc:
         current_thread.cron_at = cron_at
 
         configured_job = job_class.constantize.set(set_value)
-        kwargs_value.present? ? configured_job.perform_later(*args_value, **kwargs_value) : configured_job.perform_later(*args_value)
+        I18n.with_locale(I18n.default_locale) do
+          kwargs_value.present? ? configured_job.perform_later(*args_value, **kwargs_value) : configured_job.perform_later(*args_value)
+        end
       end
     rescue ActiveRecord::RecordNotUnique
       false

data/lib/good_job/active_job_extensions/concurrency.rb

@@ -31,11 +31,17 @@ module GoodJob
           next(block.call) if CurrentThread.active_job_id == job.job_id
 
           enqueue_limit = job.class.good_job_concurrency_config[:enqueue_limit]
-          total_limit = job.class.good_job_concurrency_config[:total_limit]
+          enqueue_limit = instance_exec(&enqueue_limit) if enqueue_limit.respond_to?(:call)
+          enqueue_limit = nil unless enqueue_limit.present? && (0...Float::INFINITY).cover?(enqueue_limit)
 
-          has_limit = (enqueue_limit.present? && (0...Float::INFINITY).cover?(enqueue_limit)) ||
-                      (total_limit.present? && (0...Float::INFINITY).cover?(total_limit))
-          next(block.call) unless has_limit
+          unless enqueue_limit
+            total_limit = job.class.good_job_concurrency_config[:total_limit]
+            total_limit = instance_exec(&total_limit) if total_limit.respond_to?(:call)
+            total_limit = nil unless total_limit.present? && (0...Float::INFINITY).cover?(total_limit)
+          end
+
+          limit = enqueue_limit || total_limit
+          next(block.call) unless limit
 
           # Only generate the concurrency key on the initial enqueue in case it is dynamic
           job.good_job_concurrency_key ||= job._good_job_concurrency_key
@@ -50,7 +56,7 @@ module GoodJob
                                   end
 
             # The job has not yet been enqueued, so check if adding it will go over the limit
-            block.call unless enqueue_concurrency + 1 > (enqueue_limit || total_limit)
+            block.call unless (enqueue_concurrency + 1) > limit
           end
         end
 
@@ -64,11 +70,18 @@ module GoodJob
           # Don't attempt to enforce concurrency limits with other queue adapters.
           next unless job.class.queue_adapter.is_a?(GoodJob::Adapter)
 
-          perform_limit = job.class.good_job_concurrency_config[:perform_limit] ||
-                          job.class.good_job_concurrency_config[:total_limit]
+          perform_limit = job.class.good_job_concurrency_config[:perform_limit]
+          perform_limit = instance_exec(&perform_limit) if perform_limit.respond_to?(:call)
+          perform_limit = nil unless perform_limit.present? && (0...Float::INFINITY).cover?(perform_limit)
+
+          unless perform_limit
+            total_limit = job.class.good_job_concurrency_config[:total_limit]
+            total_limit = instance_exec(&total_limit) if total_limit.respond_to?(:call)
+            total_limit = nil unless total_limit.present? && (0...Float::INFINITY).cover?(total_limit)
+          end
 
-          has_limit = perform_limit.present? && (0...Float::INFINITY).cover?(perform_limit)
-          next unless has_limit
+          limit = perform_limit || total_limit
+          next unless limit
 
           key = job.good_job_concurrency_key
           next if key.blank?
@@ -79,7 +92,7 @@ module GoodJob
           end
 
           GoodJob::Execution.advisory_lock_key(key, function: "pg_advisory_lock") do
-            allowed_active_job_ids = GoodJob::Execution.unfinished.where(concurrency_key: key).advisory_locked.order(Arel.sql("COALESCE(performed_at, scheduled_at, created_at) ASC")).limit(perform_limit).pluck(:active_job_id)
+            allowed_active_job_ids = GoodJob::Execution.unfinished.where(concurrency_key: key).advisory_locked.order(Arel.sql("COALESCE(performed_at, scheduled_at, created_at) ASC")).limit(limit).pluck(:active_job_id)
             # The current job has already been locked and will appear in the previous query
             raise GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError unless allowed_active_job_ids.include? job.job_id
           end

data/lib/good_job/log_subscriber.rb

@@ -32,7 +32,7 @@ module GoodJob
       return unless exception
 
       error do
-        "GoodJob error: #{exception}\n #{exception.backtrace}"
+        "GoodJob error: #{exception.class}: #{exception}\n #{exception.backtrace}"
       end
     end
 
@@ -42,7 +42,7 @@ module GoodJob
       return unless exception
 
       error do
-        "GoodJob error: #{exception}\n #{exception.backtrace}"
+        "GoodJob error: #{exception.class}: #{exception}\n #{exception.backtrace}"
       end
     end
 
@@ -123,10 +123,10 @@ module GoodJob
 
     # @!macro notification_responder
     def notifier_notify_error(event)
-      error = event.payload[:error]
+      exception = event.payload[:error]
 
       error do
-        "Notifier errored: #{error}"
+        "Notifier errored: #{exception.class}: #{exception}\n #{exception.backtrace}"
       end
     end
 

data/lib/good_job/notifier.rb

@@ -40,6 +40,7 @@ module GoodJob # :nodoc:
       PG::UnableToSend
       PG::Error
     ].freeze
+    CONNECTION_ERRORS_REPORTING_THRESHOLD = 3
 
     # @!attribute [r] instances
     #   @!scope class
@@ -70,6 +71,8 @@ module GoodJob # :nodoc:
     def initialize(*recipients)
       @recipients = Concurrent::Array.new(recipients)
       @listening = Concurrent::AtomicBoolean.new(false)
+      @connection_errors_count = Concurrent::AtomicFixnum.new(0)
+      @connection_errors_reported = Concurrent::AtomicBoolean.new(false)
 
       self.class.instances << self
 
@@ -128,7 +131,6 @@ module GoodJob # :nodoc:
     # @return [void]
     def listen_observer(_time, _result, thread_error)
       if thread_error
-        GoodJob._on_thread_error(thread_error)
         ActiveSupport::Notifications.instrument("notifier_notify_error.good_job", { error: thread_error })
 
         connection_error = CONNECTION_ERRORS.any? do |error_string|
@@ -137,6 +139,16 @@ module GoodJob # :nodoc:
 
           thread_error.is_a? error_class
         end
+
+        if connection_error
+          @connection_errors_count.increment
+          if @connection_errors_reported.false? && @connection_errors_count.value >= CONNECTION_ERRORS_REPORTING_THRESHOLD
+            GoodJob._on_thread_error(thread_error)
+            @connection_errors_reported.make_true
+          end
+        else
+          GoodJob._on_thread_error(thread_error)
+        end
       end
 
       return if shutdown?
@@ -175,6 +187,8 @@ module GoodJob # :nodoc:
                     target.send(method_name, parsed_payload)
                   end
                 end
+
+                reset_connection_errors
               end
             end
           end
@@ -223,5 +237,10 @@ module GoodJob # :nodoc:
         sleep WAIT_INTERVAL
       end
     end
+
+    def reset_connection_errors
+      @connection_errors_count.value = 0
+      @connection_errors_reported.make_false
+    end
   end
 end

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '3.4.0'
+  VERSION = '3.4.3'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 3.4.0
+  version: 3.4.3
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-08-05 00:00:00.000000000 Z
+date: 2022-08-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -290,20 +290,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: sigdump
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: yard
   requirement: !ruby/object:Gem::Requirement