good_job 1.12.1 → 1.13.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5b4629010561604c4f1d77812c3ca7301e5600fa8e0637fd584a769898d43aa5
-  data.tar.gz: 1216b9d590c40c047fa25c13afec94021891752bab8ec36ff7f6c968af8e285e
+  metadata.gz: ef04aa133c3e3f39b4e5967f61441617808d51a8d13b79e9f4e61a89c3da0c63
+  data.tar.gz: a5abe5ff64d1c7abb197b78f0aa7812834b9034fb0340dbb8abb6d2a42912af9
 SHA512:
-  metadata.gz: ef51328facf01deeec6a2e67c38c526c944d1c7381fa45cf97f7f05aa4897ac711c324f616e8d948c3c5e6e4e5fae636ea2c7c3c9d48d79de0ead78ed438580f
-  data.tar.gz: e4efa15858ac7a6c0849ec084a00de8fbb8f6751523e78c9234b118bee2c84f3709c549cfc1f161af5166b2d54ff1f358b2dbd18e54adca06d6c62ad10757827
+  metadata.gz: 6effeb75fd0d5a0541f4be984e6a0931fba2d6849b1d5061941276b5ee977b3620ce13b2328b64fdd27b1d66ab1753087aed612d7c301b7a100fc709a3740697
+  data.tar.gz: 4694a54a3f1dffd84e4875fc7169e49f6edff3baa92fc458720939738b3db6b6f907883601c480fd4aa1d64549b7a1a0352b3c2481df1ae67057685efa9414a6

data/CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## [v1.13.2](https://github.com/bensheldon/good_job/tree/v1.13.2) (2021-08-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.1...v1.13.2)
+
+**Merged pull requests:**
+
+- Add deprecation notice that `async` mode will be renamed `async_all` in GoodJob v2.0 [\#339](https://github.com/bensheldon/good_job/pull/339) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.13.1](https://github.com/bensheldon/good_job/tree/v1.13.1) (2021-08-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.0...v1.13.1)
+
+**Fixed bugs:**
+
+- Don’t attempt to enforce concurrency limits with other queue adapters [\#333](https://github.com/bensheldon/good_job/pull/333) ([codyrobbins](https://github.com/codyrobbins))
+
+## [v1.13.0](https://github.com/bensheldon/good_job/tree/v1.13.0) (2021-08-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.12.2...v1.13.0)
+
+**Implemented enhancements:**
+
+- Track if a GoodJob::Job has been subsequently retried [\#331](https://github.com/bensheldon/good_job/pull/331) ([bensheldon](https://github.com/bensheldon))
+- Wrap and truncate error message, which can be a huge text [\#294](https://github.com/bensheldon/good_job/pull/294) ([morgoth](https://github.com/morgoth))
+
+**Closed issues:**
+
+- Add hyphen to lock string. e.g. "table\_name-column" instead of "table\_namecolumn [\#334](https://github.com/bensheldon/good_job/issues/334)
+- Optimize db indexes in advance of v2.0.0 [\#332](https://github.com/bensheldon/good_job/issues/332)
+- wait\_until in development? [\#330](https://github.com/bensheldon/good_job/issues/330)
+- Race conditions in ActiveJob concurrency extension [\#325](https://github.com/bensheldon/good_job/issues/325)
+- Store in database if a job has been ActiveJob retried [\#321](https://github.com/bensheldon/good_job/issues/321)
+- Revisit and embrace concurrency control, scheduled jobs, and other extensions of ActiveJob [\#255](https://github.com/bensheldon/good_job/issues/255)
+- Why 1 million jobs per day? [\#222](https://github.com/bensheldon/good_job/issues/222)
+
+## [v1.12.2](https://github.com/bensheldon/good_job/tree/v1.12.2) (2021-08-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.12.1...v1.12.2)
+
+**Fixed bugs:**
+
+- Fixes for race conditions in ActiveJob concurrency extension [\#326](https://github.com/bensheldon/good_job/pull/326) ([codyrobbins](https://github.com/codyrobbins))
+
+**Merged pull requests:**
+
+- On gem release, add instructions to author a Github Release [\#324](https://github.com/bensheldon/good_job/pull/324) ([bensheldon](https://github.com/bensheldon))
+
 ## [v1.12.1](https://github.com/bensheldon/good_job/tree/v1.12.1) (2021-08-05)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.12.0...v1.12.1)
@@ -29,7 +76,7 @@
 - Add the ability to schedule repeating / recurring / cron-like jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
 - Add cron-like support for recurring/repeating jobs [\#297](https://github.com/bensheldon/good_job/pull/297) ([bensheldon](https://github.com/bensheldon))
 
-**Merged pull requests:**
+**Fixed bugs:**
 
 - Place Dashboard shared view partials under `good_job` namespace [\#310](https://github.com/bensheldon/good_job/pull/310) ([bensheldon](https://github.com/bensheldon))
 - Ensure Dashboard inline javascript has CSP nonce for strict Content-Security Policy [\#309](https://github.com/bensheldon/good_job/pull/309) ([bensheldon](https://github.com/bensheldon))

data/README.md

@@ -10,7 +10,7 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
 - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
 - **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
-- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
+- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue 1-million jobs/day and more.
 
 For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
 

data/engine/app/views/good_job/shared/_jobs_table.erb

@@ -19,7 +19,7 @@
             <td><%= job.serialized_params['job_class'] %></td>
             <td><%= job.queue_name %></td>
             <td><%= job.scheduled_at || job.created_at %></td>
-            <td><%= job.error %></td>
+            <td class="text-break"><%= truncate(job.error, length: 1_000) %></td>
             <td>
               <%= tag.button "Preview", type: "button", class: "btn btn-sm btn-outline-primary", role: "button",
                 data: {bs_toggle: "collapse", bs_target: "##{dom_id(job, 'params')}"},

data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb

@@ -16,6 +16,7 @@ class CreateGoodJobs < ActiveRecord::Migration[5.2]
       t.uuid :active_job_id
       t.text :concurrency_key
       t.text :cron_key
+      t.uuid :retried_good_job_id
     end
 
     add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)", name: "index_good_jobs_on_scheduled_at"

data/lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+class AddRetriedGoodJobIdToGoodJobs < ActiveRecord::Migration[5.2]
+  def change
+    reversible do |dir|
+      dir.up do
+        # Ensure this incremental update migration is idempotent
+        # with monolithic install migration.
+        return if connection.column_exists?(:good_jobs, :retried_good_job_id)
+      end
+    end
+
+    add_column :good_jobs, :retried_good_job_id, :uuid
+  end
+end

data/lib/good_job/active_job_extensions/concurrency.rb

@@ -9,21 +9,24 @@ module GoodJob
       included do
         class_attribute :good_job_concurrency_config, instance_accessor: false, default: {}
 
-        before_enqueue do |job|
+        around_enqueue do |job, block|
+          # Don't attempt to enforce concurrency limits with other queue adapters.
+          next(block.call) unless job.class.queue_adapter.is_a?(GoodJob::Adapter)
+
           # Always allow jobs to be retried because the current job's execution will complete momentarily
-          next if CurrentExecution.active_job_id == job.job_id
+          next(block.call) if CurrentExecution.active_job_id == job.job_id
 
           limit = job.class.good_job_concurrency_config.fetch(:enqueue_limit, Float::INFINITY)
-          next if limit.blank? || (0...Float::INFINITY).exclude?(limit)
+          next(block.call) if limit.blank? || (0...Float::INFINITY).exclude?(limit)
 
           key = job.good_job_concurrency_key
-          next if key.blank?
+          next(block.call) if key.blank?
 
           GoodJob::Job.new.with_advisory_lock(key: key, function: "pg_advisory_lock") do
             # TODO: Why is `unscoped` necessary? Nested scope is bleeding into subsequent query?
             enqueue_concurrency = GoodJob::Job.unscoped.where(concurrency_key: key).unfinished.count
             # The job has not yet been enqueued, so check if adding it will go over the limit
-            throw :abort if enqueue_concurrency + 1 > limit
+            block.call unless enqueue_concurrency + 1 > limit
           end
         end
 
@@ -34,6 +37,9 @@ module GoodJob
         )
 
         before_perform do |job|
+          # Don't attempt to enforce concurrency limits with other queue adapters.
+          next unless job.class.queue_adapter.is_a?(GoodJob::Adapter)
+
           limit = job.class.good_job_concurrency_config.fetch(:perform_limit, Float::INFINITY)
           next if limit.blank? || (0...Float::INFINITY).exclude?(limit)
 
@@ -41,9 +47,9 @@ module GoodJob
           next if key.blank?
 
           GoodJob::Job.new.with_advisory_lock(key: key, function: "pg_advisory_lock") do
-            perform_concurrency = GoodJob::Job.unscoped.where(concurrency_key: key).advisory_locked.count
+            allowed_active_job_ids = GoodJob::Job.unscoped.where(concurrency_key: key).advisory_locked.order(Arel.sql("COALESCE(performed_at, scheduled_at, created_at) ASC")).limit(limit).pluck(:active_job_id)
             # The current job has already been locked and will appear in the previous query
-            raise GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError if perform_concurrency > limit
+            raise GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError unless allowed_active_job_ids.include? job.job_id
           end
         end
       end

data/lib/good_job/adapter.rb

@@ -130,7 +130,7 @@ module GoodJob
     # Whether in +:async+ execution mode.
     # @return [Boolean]
     def execute_async?
-      @configuration.execution_mode == :async ||
+      @configuration.execution_mode.in?([:async, :async_all]) ||
         @configuration.execution_mode == :async_server && in_server_process?
     end
 

data/lib/good_job/configuration.rb

@@ -7,7 +7,7 @@ module GoodJob
   #
   class Configuration
     # Valid execution modes.
-    EXECUTION_MODES = [:async, :async_server, :external, :inline].freeze
+    EXECUTION_MODES = [:async, :async_all, :async_server, :external, :inline].freeze
     # Default number of threads to use per {Scheduler}
     DEFAULT_MAX_THREADS = 5
     # Default number of seconds between polls for jobs
@@ -58,7 +58,18 @@ module GoodJob
                end
 
         if mode
-          mode.to_sym
+          mode_sym = mode.to_sym
+          if mode_sym == :async
+            ActiveSupport::Deprecation.warn <<~DEPRECATION
+              The next major version of GoodJob will redefine the meaning of 'async'
+              execution mode to be equivalent to 'async_server' and only execute
+              within the webserver process.
+
+              To continue using the v1.0 semantics of 'async', use `async_all` instead.
+
+            DEPRECATION
+          end
+          mode_sym
         elsif Rails.env.development? || Rails.env.test?
           :inline
         else

data/lib/good_job/current_execution.rb

@@ -5,12 +5,6 @@ module GoodJob
   # Thread-local attributes for passing values from Instrumentation.
   # (Cannot use ActiveSupport::CurrentAttributes because ActiveJob resets it)
   module CurrentExecution
-    # @!attribute [rw] active_job_id
-    #   @!scope class
-    #   ActiveJob ID
-    #   @return [String, nil]
-    thread_mattr_accessor :active_job_id
-
     # @!attribute [rw] cron_key
     #   @!scope class
     #   Cron Key
@@ -29,14 +23,26 @@ module GoodJob
     #   @return [Exception, nil]
     thread_mattr_accessor :error_on_retry
 
+    # @!attribute [rw] good_job
+    #   @!scope class
+    #   Cron Key
+    #   @return [GoodJob::Job, nil]
+    thread_mattr_accessor :good_job
+
     # Resets attributes
     # @return [void]
     def self.reset
-      self.active_job_id = nil
+      self.cron_key = nil
+      self.good_job = nil
       self.error_on_discard = nil
       self.error_on_retry = nil
     end
 
+    # @return [String] UUID of the currently executing GoodJob::Job
+    def self.active_job_id
+      good_job&.active_job_id
+    end
+
     # @return [Integer] Current process ID
     def self.process_id
       Process.pid

data/lib/good_job/job.rb

@@ -52,6 +52,20 @@ module GoodJob
       end
     end
 
+    def self._migration_pending_warning
+      ActiveSupport::Deprecation.warn(<<~DEPRECATION)
+        GoodJob has pending database migrations. To create the migration files, run:
+
+            rails generate good_job:update
+
+        To apply the migration files, run:
+
+            rails db:migrate
+
+      DEPRECATION
+      nil
+    end
+
     # Get Jobs with given class name
     # @!method with_job_class
     # @!scope class
@@ -110,6 +124,18 @@ module GoodJob
     # @return [ActiveRecord::Relation]
     scope :running, -> { where.not(performed_at: nil).where(finished_at: nil) }
 
+    # Get Jobs that do not have subsequent retries
+    # @!method running
+    # @!scope class
+    # @return [ActiveRecord::Relation]
+    scope :head, -> { where(retried_good_job_id: nil) }
+
+    # Get Jobs have errored that will not be retried further
+    # @!method running
+    # @!scope class
+    # @return [ActiveRecord::Relation]
+    scope :dead, -> { head.where.not(error: nil) }
+
     # Get Jobs on queues that match the given queue string.
     # @!method queue_string(string)
     # @!scope class
@@ -199,7 +225,6 @@ module GoodJob
     def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
       ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
         good_job_args = {
-          cron_key: CurrentExecution.cron_key,
           queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
           priority: active_job.priority || DEFAULT_PRIORITY,
           serialized_params: active_job.serialize,
@@ -210,31 +235,23 @@ module GoodJob
         if column_names.include?('active_job_id')
           good_job_args[:active_job_id] = active_job.job_id
         else
-          ActiveSupport::Deprecation.warn(<<~DEPRECATION)
-            GoodJob has pending database migrations. To create the migration files, run:
-
-                rails generate good_job:update
-
-            To apply the migration files, run:
-
-                rails db:migrate
-
-          DEPRECATION
+          _migration_pending_warning
         end
 
         if column_names.include?('concurrency_key')
           good_job_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
         else
-          ActiveSupport::Deprecation.warn(<<~DEPRECATION)
-            GoodJob has pending database migrations. To create the migration files, run:
-
-                rails generate good_job:update
-
-            To apply the migration files, run:
-
-                rails db:migrate
+          _migration_pending_warning
+        end
 
-          DEPRECATION
+        if column_names.include?('cron_key')
+          if CurrentExecution.cron_key
+            good_job_args[:cron_key] = CurrentExecution.cron_key
+          elsif CurrentExecution.active_job_id == active_job.job_id
+            good_job_args[:cron_key] = CurrentExecution.good_job.cron_key
+          end
+        else
+          _migration_pending_warning
         end
 
         good_job = GoodJob::Job.new(**good_job_args)
@@ -244,6 +261,12 @@ module GoodJob
         good_job.save!
         active_job.provider_job_id = good_job.id
 
+        if column_names.include?('retried_good_job_id')
+          CurrentExecution.good_job.retried_good_job_id = good_job.id if CurrentExecution.good_job && CurrentExecution.good_job.active_job_id == active_job.job_id
+        else
+          _migration_pending_warning
+        end
+
         good_job
       end
     end
@@ -283,24 +306,19 @@ module GoodJob
     end
 
     def active_job_id
-      super || serialized_params['job_id']
+      if self.class.column_names.include?('active_job_id')
+        super
+      else
+        self.class._migration_pending_warning
+        serialized_params['job_id']
+      end
     end
 
     def cron_key
       if self.class.column_names.include?('cron_key')
         super
       else
-        ActiveSupport::Deprecation.warn(<<~DEPRECATION)
-          GoodJob has pending database migrations. To create the migration files, run:
-
-              rails generate good_job:update
-
-          To apply the migration files, run:
-
-              rails db:migrate
-
-        DEPRECATION
-
+        self.class._migration_pending_warning
         nil
       end
     end
@@ -314,8 +332,7 @@ module GoodJob
       )
 
       GoodJob::CurrentExecution.reset
-      GoodJob::CurrentExecution.active_job_id = active_job_id
-      GoodJob::CurrentExecution.cron_key = cron_key
+      GoodJob::CurrentExecution.good_job = self
       ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self, process_id: GoodJob::CurrentExecution.process_id, thread_name: GoodJob::CurrentExecution.thread_name }) do
         value = ActiveJob::Base.execute(params)
 

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '1.12.1'
+  VERSION = '1.13.2'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.12.1
+  version: 1.13.2
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-08-05 00:00:00.000000000 Z
+date: 2021-08-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -374,6 +374,7 @@ files:
 - lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb
 - lib/generators/good_job/templates/update/migrations/02_add_active_job_id_concurrency_key_cron_key_to_good_jobs.rb
 - lib/generators/good_job/templates/update/migrations/03_add_active_job_id_index_and_concurrency_key_index_to_good_jobs.rb
+- lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb
 - lib/generators/good_job/update_generator.rb
 - lib/good_job.rb
 - lib/good_job/active_job_extensions.rb