good_job 1.12.2 → 1.99.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0af2c2293c2a0970a95ac200776850daa396a91d75aeda3f4612de23ee951a5c
-  data.tar.gz: c938c0efca849771e68b39634e56fbdacf2987a83bdf8115e8050b9c49da40c3
+  metadata.gz: fe59c2b49eaef9a76ecbf1af80c6ea765fb89a5d61f01b0436595e15e377019e
+  data.tar.gz: 15c0ee082903f8316de7656e855489492a00e90b3e75e55475f59824b49c0542
 SHA512:
-  metadata.gz: 3b4da4eec468271fc88260fc0abcdb6faaaa9e43532e5587338fdca0a4f92f89d4e4f780678fe2c74a714adb0e7a1af263fb0578193382ee8181e11c44aea20d
-  data.tar.gz: 91f4ae034471774f5ec1915750ccbf1d9d6ad9cc68df4f5ffa4c0bf6bca57928115e6871a59d30d6bab79b22faa7a4a293d55f55c1bf2cc659d55c0c60049040
+  metadata.gz: 22c7de0952050994cb220cda916ce85187775d2b79692b8ee094a0e6e76f84945ad528327fc3e83e50263e46272094949d2d6270a0332c19e8f79fb0841e0d80
+  data.tar.gz: '0229a23385ed171a50a39d54ab5295a5147110d203e84518643caa49c2dab16eee16263b54c86245a376e946402e9d39d52f05c3cde6ba2906f8f32fd77046e0'

data/CHANGELOG.md

@@ -1,5 +1,53 @@
 # Changelog
 
+## [v1.99.0](https://github.com/bensheldon/good_job/tree/v1.99.0) (2021-08-24)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.2...v1.99.0)
+
+**Closed issues:**
+
+- Set Advisory Lock on ActiveJob job uuid instead of GoodJob's job uuid [\#272](https://github.com/bensheldon/good_job/issues/272)
+
+**Merged pull requests:**
+
+- Add upgrade instructions for v1 to v2 [\#345](https://github.com/bensheldon/good_job/pull/345) ([bensheldon](https://github.com/bensheldon))
+- Add transitional/temporary additional lock on good\_jobs-\[active\_job\_id\] [\#336](https://github.com/bensheldon/good_job/pull/336) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.13.2](https://github.com/bensheldon/good_job/tree/v1.13.2) (2021-08-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.1...v1.13.2)
+
+**Merged pull requests:**
+
+- Add deprecation notice that `async` mode will be renamed `async_all` in GoodJob v2.0 [\#339](https://github.com/bensheldon/good_job/pull/339) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.13.1](https://github.com/bensheldon/good_job/tree/v1.13.1) (2021-08-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.13.0...v1.13.1)
+
+**Fixed bugs:**
+
+- Don’t attempt to enforce concurrency limits with other queue adapters [\#333](https://github.com/bensheldon/good_job/pull/333) ([codyrobbins](https://github.com/codyrobbins))
+
+## [v1.13.0](https://github.com/bensheldon/good_job/tree/v1.13.0) (2021-08-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.12.2...v1.13.0)
+
+**Implemented enhancements:**
+
+- Track if a GoodJob::Job has been subsequently retried [\#331](https://github.com/bensheldon/good_job/pull/331) ([bensheldon](https://github.com/bensheldon))
+- Wrap and truncate error message, which can be a huge text [\#294](https://github.com/bensheldon/good_job/pull/294) ([morgoth](https://github.com/morgoth))
+
+**Closed issues:**
+
+- Add hyphen to lock string. e.g. "table\_name-column" instead of "table\_namecolumn [\#334](https://github.com/bensheldon/good_job/issues/334)
+- Optimize db indexes in advance of v2.0.0 [\#332](https://github.com/bensheldon/good_job/issues/332)
+- wait\_until in development? [\#330](https://github.com/bensheldon/good_job/issues/330)
+- Race conditions in ActiveJob concurrency extension [\#325](https://github.com/bensheldon/good_job/issues/325)
+- Store in database if a job has been ActiveJob retried [\#321](https://github.com/bensheldon/good_job/issues/321)
+- Revisit and embrace concurrency control, scheduled jobs, and other extensions of ActiveJob [\#255](https://github.com/bensheldon/good_job/issues/255)
+- Why 1 million jobs per day? [\#222](https://github.com/bensheldon/good_job/issues/222)
+
 ## [v1.12.2](https://github.com/bensheldon/good_job/tree/v1.12.2) (2021-08-13)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.12.1...v1.12.2)

data/README.md

@@ -10,7 +10,7 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
 - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
 - **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
-- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
+- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue 1-million jobs/day and more.
 
 For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
 
@@ -41,6 +41,8 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [ActiveJob concurrency](#activejob-concurrency)
     - [Cron-style repeating/recurring jobs](#cron-style-repeatingrecurring-jobs)
     - [Updating](#updating)
+        - [Upgrading minor versions](#upgrading-minor-versions)
+        - [Upgrading v1 to v2](#upgrading-v1-to-v2)
 - [Go deeper](#go-deeper)
     - [Exceptions, retries, and reliability](#exceptions-retries-and-reliability)
         - [Exceptions](#exceptions)
@@ -405,17 +407,45 @@ config.good_job.cron = {
 
 GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.
 
-To apply updates:
+#### Upgrading minor versions
 
-```bash
-bin/rails g good_job:update
-```
+Upgrading between minor versions (e.g. v1.4 to v1.5) should not introduce breaking changes, but can introduce new deprecation warnings and database migration notices.
 
-...and run the resulting migration:
+To perform upgrades to the GoodJob database tables:
 
-```bash
-bin/rails db:migrate
-```
+1. Generate new database migration files:
+
+    ```bash
+    bin/rails g good_job:update
+    ```
+
+1. Run the database migration locally
+
+    ```bash
+    bin/rails db:migrate
+    ```
+
+1. Commit the migration files and resulting `db/schema.rb` changes.
+1. Deploy the code, run the migrations against the production database, and restart server/worker processes.
+
+#### Upgrading v1 to v2
+
+GoodJob v2 introduces a new Advisory Lock key format that is different than the v1 advisory lock key format; it's therefore necessary to perform a simple, but staged production upgrade. If you are already using `>= v1.12+` no other changes are likely44
+necessary.
+
+1. Upgrade your production environment to `v1.99.x` following the minor version upgrade process, including database migrations. `v1.99` is a transitional release that is safely compatible with both `v1.x` and `v2.0.0` because it uses both `v1`- and `v2`-formatted advisory locks.
+1. Address any deprecation warnings generated by `v1.99`.
+1. Upgrade your production environment to `v1.99.x` to `v2.0.x` again following the _minor_ upgrade process.
+
+Notable changes:
+
+- Renames `:async_server` execution mode to `:async`; renames prior `:async` execution mode to `:async_all`.
+- Sets default Development environment's execution mode to `:async` with disabled polling.
+- Triggers `GoodJob.on_thread_error` for unhandled ActiveJob exceptions.
+- Renames `GoodJob.reperform_jobs_on_standard_error` accessor to `GoodJob.retry_on_unhandled_error`.
+- Renames `GoodJob::Adapter.shutdown(wait:)` argument to `GoodJob::Adapter.shutdown(timeout:)`.
+- Changes Advisory Lock key format from `good_jobs[ROW_ID]` to `good_jobs-[ACTIVE_JOB_ID]`.
+- Expects presence of columns `good_jobs.active_job_id`, `good_jobs.concurrency_key`, `good_jobs.concurrency_key`, and `good_jobs.retried_good_job_id`.
 
 ## Go deeper
 

data/engine/app/views/good_job/shared/_jobs_table.erb

@@ -19,7 +19,7 @@
             <td><%= job.serialized_params['job_class'] %></td>
             <td><%= job.queue_name %></td>
             <td><%= job.scheduled_at || job.created_at %></td>
-            <td><%= job.error %></td>
+            <td class="text-break"><%= truncate(job.error, length: 1_000) %></td>
             <td>
               <%= tag.button "Preview", type: "button", class: "btn btn-sm btn-outline-primary", role: "button",
                 data: {bs_toggle: "collapse", bs_target: "##{dom_id(job, 'params')}"},

data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb

@@ -16,6 +16,7 @@ class CreateGoodJobs < ActiveRecord::Migration[5.2]
       t.uuid :active_job_id
       t.text :concurrency_key
       t.text :cron_key
+      t.uuid :retried_good_job_id
     end
 
     add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)", name: "index_good_jobs_on_scheduled_at"

data/lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+class AddRetriedGoodJobIdToGoodJobs < ActiveRecord::Migration[5.2]
+  def change
+    reversible do |dir|
+      dir.up do
+        # Ensure this incremental update migration is idempotent
+        # with monolithic install migration.
+        return if connection.column_exists?(:good_jobs, :retried_good_job_id)
+      end
+    end
+
+    add_column :good_jobs, :retried_good_job_id, :uuid
+  end
+end

data/lib/good_job/active_job_extensions/concurrency.rb

@@ -10,6 +10,9 @@ module GoodJob
         class_attribute :good_job_concurrency_config, instance_accessor: false, default: {}
 
         around_enqueue do |job, block|
+          # Don't attempt to enforce concurrency limits with other queue adapters.
+          next(block.call) unless job.class.queue_adapter.is_a?(GoodJob::Adapter)
+
           # Always allow jobs to be retried because the current job's execution will complete momentarily
           next(block.call) if CurrentExecution.active_job_id == job.job_id
 
@@ -34,6 +37,9 @@ module GoodJob
         )
 
         before_perform do |job|
+          # Don't attempt to enforce concurrency limits with other queue adapters.
+          next unless job.class.queue_adapter.is_a?(GoodJob::Adapter)
+
           limit = job.class.good_job_concurrency_config.fetch(:perform_limit, Float::INFINITY)
           next if limit.blank? || (0...Float::INFINITY).exclude?(limit)
 

data/lib/good_job/adapter.rb

@@ -130,7 +130,7 @@ module GoodJob
     # Whether in +:async+ execution mode.
     # @return [Boolean]
     def execute_async?
-      @configuration.execution_mode == :async ||
+      @configuration.execution_mode.in?([:async, :async_all]) ||
         @configuration.execution_mode == :async_server && in_server_process?
     end
 

data/lib/good_job/configuration.rb

@@ -7,7 +7,7 @@ module GoodJob
   #
   class Configuration
     # Valid execution modes.
-    EXECUTION_MODES = [:async, :async_server, :external, :inline].freeze
+    EXECUTION_MODES = [:async, :async_all, :async_server, :external, :inline].freeze
     # Default number of threads to use per {Scheduler}
     DEFAULT_MAX_THREADS = 5
     # Default number of seconds between polls for jobs
@@ -58,7 +58,18 @@ module GoodJob
                end
 
         if mode
-          mode.to_sym
+          mode_sym = mode.to_sym
+          if mode_sym == :async
+            ActiveSupport::Deprecation.warn <<~DEPRECATION
+              The next major version of GoodJob will redefine the meaning of 'async'
+              execution mode to be equivalent to 'async_server' and only execute
+              within the webserver process.
+
+              To continue using the v1.0 semantics of 'async', use `async_all` instead.
+
+            DEPRECATION
+          end
+          mode_sym
         elsif Rails.env.development? || Rails.env.test?
           :inline
         else

data/lib/good_job/current_execution.rb

@@ -5,12 +5,6 @@ module GoodJob
   # Thread-local attributes for passing values from Instrumentation.
   # (Cannot use ActiveSupport::CurrentAttributes because ActiveJob resets it)
   module CurrentExecution
-    # @!attribute [rw] active_job_id
-    #   @!scope class
-    #   ActiveJob ID
-    #   @return [String, nil]
-    thread_mattr_accessor :active_job_id
-
     # @!attribute [rw] cron_key
     #   @!scope class
     #   Cron Key
@@ -29,14 +23,26 @@ module GoodJob
     #   @return [Exception, nil]
     thread_mattr_accessor :error_on_retry
 
+    # @!attribute [rw] good_job
+    #   @!scope class
+    #   Cron Key
+    #   @return [GoodJob::Job, nil]
+    thread_mattr_accessor :good_job
+
     # Resets attributes
     # @return [void]
     def self.reset
-      self.active_job_id = nil
+      self.cron_key = nil
+      self.good_job = nil
       self.error_on_discard = nil
       self.error_on_retry = nil
     end
 
+    # @return [String] UUID of the currently executing GoodJob::Job
+    def self.active_job_id
+      good_job&.active_job_id
+    end
+
     # @return [Integer] Current process ID
     def self.process_id
       Process.pid

data/lib/good_job/job.rb

@@ -52,6 +52,20 @@ module GoodJob
       end
     end
 
+    def self._migration_pending_warning
+      ActiveSupport::Deprecation.warn(<<~DEPRECATION)
+        GoodJob has pending database migrations. To create the migration files, run:
+
+            rails generate good_job:update
+
+        To apply the migration files, run:
+
+            rails db:migrate
+
+      DEPRECATION
+      nil
+    end
+
     # Get Jobs with given class name
     # @!method with_job_class
     # @!scope class
@@ -110,6 +124,18 @@ module GoodJob
     # @return [ActiveRecord::Relation]
     scope :running, -> { where.not(performed_at: nil).where(finished_at: nil) }
 
+    # Get Jobs that do not have subsequent retries
+    # @!method running
+    # @!scope class
+    # @return [ActiveRecord::Relation]
+    scope :head, -> { where(retried_good_job_id: nil) }
+
+    # Get Jobs have errored that will not be retried further
+    # @!method running
+    # @!scope class
+    # @return [ActiveRecord::Relation]
+    scope :dead, -> { head.where.not(error: nil) }
+
     # Get Jobs on queues that match the given queue string.
     # @!method queue_string(string)
     # @!scope class
@@ -163,7 +189,13 @@ module GoodJob
         break if good_job.blank?
         break :unlocked unless good_job&.executable?
 
-        good_job.perform
+        begin
+          good_job.with_advisory_lock(key: "good_jobs-#{good_job.active_job_id}") do
+            good_job.perform
+          end
+        rescue RecordAlreadyAdvisoryLockedError => e
+          ExecutionResult.new(value: nil, handled_error: e)
+        end
       end
     end
 
@@ -199,7 +231,6 @@ module GoodJob
     def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
       ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
         good_job_args = {
-          cron_key: CurrentExecution.cron_key,
           queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
           priority: active_job.priority || DEFAULT_PRIORITY,
           serialized_params: active_job.serialize,
@@ -210,31 +241,23 @@ module GoodJob
         if column_names.include?('active_job_id')
           good_job_args[:active_job_id] = active_job.job_id
         else
-          ActiveSupport::Deprecation.warn(<<~DEPRECATION)
-            GoodJob has pending database migrations. To create the migration files, run:
-
-                rails generate good_job:update
-
-            To apply the migration files, run:
-
-                rails db:migrate
-
-          DEPRECATION
+          _migration_pending_warning
         end
 
         if column_names.include?('concurrency_key')
           good_job_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
         else
-          ActiveSupport::Deprecation.warn(<<~DEPRECATION)
-            GoodJob has pending database migrations. To create the migration files, run:
-
-                rails generate good_job:update
-
-            To apply the migration files, run:
-
-                rails db:migrate
+          _migration_pending_warning
+        end
 
-          DEPRECATION
+        if column_names.include?('cron_key')
+          if CurrentExecution.cron_key
+            good_job_args[:cron_key] = CurrentExecution.cron_key
+          elsif CurrentExecution.active_job_id == active_job.job_id
+            good_job_args[:cron_key] = CurrentExecution.good_job.cron_key
+          end
+        else
+          _migration_pending_warning
         end
 
         good_job = GoodJob::Job.new(**good_job_args)
@@ -244,6 +267,12 @@ module GoodJob
         good_job.save!
         active_job.provider_job_id = good_job.id
 
+        if column_names.include?('retried_good_job_id')
+          CurrentExecution.good_job.retried_good_job_id = good_job.id if CurrentExecution.good_job && CurrentExecution.good_job.active_job_id == active_job.job_id
+        else
+          _migration_pending_warning
+        end
+
         good_job
       end
     end
@@ -283,24 +312,19 @@ module GoodJob
     end
 
     def active_job_id
-      super || serialized_params['job_id']
+      if self.class.column_names.include?('active_job_id')
+        super
+      else
+        self.class._migration_pending_warning
+        serialized_params['job_id']
+      end
     end
 
     def cron_key
       if self.class.column_names.include?('cron_key')
         super
       else
-        ActiveSupport::Deprecation.warn(<<~DEPRECATION)
-          GoodJob has pending database migrations. To create the migration files, run:
-
-              rails generate good_job:update
-
-          To apply the migration files, run:
-
-              rails db:migrate
-
-        DEPRECATION
-
+        self.class._migration_pending_warning
         nil
       end
     end
@@ -314,8 +338,7 @@ module GoodJob
       )
 
       GoodJob::CurrentExecution.reset
-      GoodJob::CurrentExecution.active_job_id = active_job_id
-      GoodJob::CurrentExecution.cron_key = cron_key
+      GoodJob::CurrentExecution.good_job = self
       ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self, process_id: GoodJob::CurrentExecution.process_id, thread_name: GoodJob::CurrentExecution.thread_name }) do
         value = ActiveJob::Base.execute(params)
 

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '1.12.2'
+  VERSION = '1.99.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.12.2
+  version: 1.99.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-08-13 00:00:00.000000000 Z
+date: 2021-08-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -374,6 +374,7 @@ files:
 - lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb
 - lib/generators/good_job/templates/update/migrations/02_add_active_job_id_concurrency_key_cron_key_to_good_jobs.rb
 - lib/generators/good_job/templates/update/migrations/03_add_active_job_id_index_and_concurrency_key_index_to_good_jobs.rb
+- lib/generators/good_job/templates/update/migrations/04_add_retried_good_job_id_to_good_jobs.rb
 - lib/generators/good_job/update_generator.rb
 - lib/good_job.rb
 - lib/good_job/active_job_extensions.rb