good_job 1.10.0 → 1.11.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 96c16ec6a03de5eaf957327ad3f9161d19665eeef2e8bc300caa758a84cfe58f
-  data.tar.gz: 22be95e4ae3d2dc29a8b242cb37889d21d2b19f119184a5a33d07f04e9a3b453
+  metadata.gz: d78c670393e4cddbedfe9512fdda95855709d830618cd63cfde7e724f26ce6c8
+  data.tar.gz: 494d3b0d6ec2b05d0bcc3386c14baca6071635647ed808a9b3543b5f1ebc4bfb
 SHA512:
-  metadata.gz: a59d51fdaf7e08a551e6401bdc6fb59d56e8397bc15b7f6ead69ce5eb45da653c149ebad6583af739cdcb619c2d827fb19f50becf4e4b3780ef4d31dc9be4a50
-  data.tar.gz: a9c3c96e76c19270b436d03fcff64e2e873cb6f14b784cbe8b2dd20e2deee455ce2ae6e1869a67b812f414880a7e264a50afd2951d80b41f4ddb9acdab8a0764
+  metadata.gz: bef7307febc465eab519c44a214287d5203bf0a024d83e46160dbbaf94b3a0fcd1317fbe99e446d9b0f01fb2bc8845a6dfe722fb123020aab96fe31b0c189d71
+  data.tar.gz: 737c03b9ad35169c9db43070abca6f644816876217fd333831db568f43eca6738dd9327eb7a403eb901ab6643e06445ba7764453d3f9b05e9e7c1159f7ba0963

data/CHANGELOG.md

@@ -1,5 +1,60 @@
 # Changelog
 
+## [v1.11.2](https://github.com/bensheldon/good_job/tree/v1.11.2) (2021-07-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.11.1...v1.11.2)
+
+**Fixed bugs:**
+
+- Notifier waits to retry listening when database is unavailable [\#301](https://github.com/bensheldon/good_job/pull/301) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Handle database connection drops [\#296](https://github.com/bensheldon/good_job/issues/296)
+- Using the `async` worker results in `ActiveModel::UnknownAttributeError  unknown attribute 'create_with_advisory_lock' for GoodJob::Job`. [\#290](https://github.com/bensheldon/good_job/issues/290)
+
+**Merged pull requests:**
+
+- Rename development and test databases to be `good_job` [\#300](https://github.com/bensheldon/good_job/pull/300) ([bensheldon](https://github.com/bensheldon))
+- Move generators spec into top-level spec directory; update dependencies [\#299](https://github.com/bensheldon/good_job/pull/299) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.11.1](https://github.com/bensheldon/good_job/tree/v1.11.1) (2021-07-07)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.11.0...v1.11.1)
+
+**Closed issues:**
+
+- Database connection required while loading the code on 1.10.x [\#291](https://github.com/bensheldon/good_job/issues/291)
+
+**Merged pull requests:**
+
+- Defer accessing ActiveRecord `primary_key` in Lockable [\#293](https://github.com/bensheldon/good_job/pull/293) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.11.0](https://github.com/bensheldon/good_job/tree/v1.11.0) (2021-07-07)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.10.1...v1.11.0)
+
+**Implemented enhancements:**
+
+- Add concurrency extension for ActiveJob [\#281](https://github.com/bensheldon/good_job/pull/281) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Investigate GoodJob concurrency [\#289](https://github.com/bensheldon/good_job/issues/289)
+- Problem with migrating database on 1.10.0 [\#287](https://github.com/bensheldon/good_job/issues/287)
+- Support migration --database option for install task? [\#267](https://github.com/bensheldon/good_job/issues/267)
+- Add GoodJob to Ruby Toolbox [\#243](https://github.com/bensheldon/good_job/issues/243)
+- Custom advisory locks to prevent certain jobs from being worked on concurrently? [\#206](https://github.com/bensheldon/good_job/issues/206)
+
+## [v1.10.1](https://github.com/bensheldon/good_job/tree/v1.10.1) (2021-06-30)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.10.0...v1.10.1)
+
+**Merged pull requests:**
+
+- Remove `FOR UPDATE SKIP LOCKED` from job locking sql statement [\#288](https://github.com/bensheldon/good_job/pull/288) ([bensheldon](https://github.com/bensheldon))
+- Update GH Test Matrix with latest JRuby 9.2.19.0 [\#283](https://github.com/bensheldon/good_job/pull/283) ([tedhexaflow](https://github.com/tedhexaflow))
+
 ## [v1.10.0](https://github.com/bensheldon/good_job/tree/v1.10.0) (2021-06-29)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.9.6...v1.10.0)

data/README.md

@@ -38,6 +38,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Configuration options](#configuration-options)
     - [Global options](#global-options)
     - [Dashboard](#dashboard)
+    - [ActiveJob Concurrency](#activejob-concurrency)
     - [Updating](#updating)
 - [Go deeper](#go-deeper)
     - [Exceptions, retries, and reliability](#exceptions-retries-and-reliability)
@@ -319,6 +320,35 @@ GoodJob includes a Dashboard as a mountable `Rails::Engine`.
     end
     ```
 
+### ActiveJob Concurrency
+
+GoodJob can extend ActiveJob to provide limits on concurrently running jobs, either at time of _enqueue_ or at _perform_.
+
+**Note:** Limiting concurrency at _enqueue_ requires Rails 6.0+ because Rails 5.2 does not support `throw :abort` in ActiveJob callbacks.
+
+```ruby
+class MyJob < ApplicationJob
+  include GoodJob::ActiveJobExtensions::Concurrency
+
+  good_job_control_concurrency_with(
+    # Maximum number of jobs with the concurrency key to be concurrently enqueued
+    enqueue_limit: 2,
+
+    # Maximum number of jobs with the concurrency key to be concurrently performed
+    perform_limit: 1,
+
+    # A unique key to be globally locked against.
+    # Can be String or Lambda/Proc that is invoked in the context of the job.
+    # Note: Arguments passed to #perform_later must be accessed through `arguments` method.
+    key: -> { "Unique-#{arguments.first}" } #  MyJob.perform_later("Alice") => "Unique-Alice"
+  )
+
+  def perform(first_name)
+    # do work
+  end
+end
+```
+
 ### Updating
 
 GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.

data/lib/good_job/active_job_extensions.rb

@@ -0,0 +1,4 @@
+module GoodJob
+  module ActiveJobExtensions
+  end
+end

data/lib/good_job/active_job_extensions/concurrency.rb

@@ -0,0 +1,68 @@
+module GoodJob
+  module ActiveJobExtensions
+    module Concurrency
+      extend ActiveSupport::Concern
+
+      ConcurrencyExceededError = Class.new(StandardError)
+
+      included do
+        class_attribute :good_job_concurrency_config, instance_accessor: false, default: {}
+
+        before_enqueue do |job|
+          # Always allow jobs to be retried because the current job's execution will complete momentarily
+          next if CurrentExecution.active_job_id == job.job_id
+
+          limit = job.class.good_job_concurrency_config.fetch(:enqueue_limit, Float::INFINITY)
+          next if limit.blank? || (0...Float::INFINITY).exclude?(limit)
+
+          key = job.good_job_concurrency_key
+          next if key.blank?
+
+          GoodJob::Job.new.with_advisory_lock(key: key, function: "pg_advisory_lock") do
+            # TODO: Why is `unscoped` necessary? Nested scope is bleeding into subsequent query?
+            enqueue_concurrency = GoodJob::Job.unscoped.where(concurrency_key: key).unfinished.count
+            # The job has not yet been enqueued, so check if adding it will go over the limit
+            throw :abort if enqueue_concurrency + 1 > limit
+          end
+        end
+
+        retry_on(
+          GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError,
+          attempts: Float::INFINITY,
+          wait: :exponentially_longer
+        )
+
+        before_perform do |job|
+          limit = job.class.good_job_concurrency_config.fetch(:perform_limit, Float::INFINITY)
+          next if limit.blank? || (0...Float::INFINITY).exclude?(limit)
+
+          key = job.good_job_concurrency_key
+          next if key.blank?
+
+          GoodJob::Job.new.with_advisory_lock(key: key, function: "pg_advisory_lock") do
+            perform_concurrency = GoodJob::Job.unscoped.where(concurrency_key: key).advisory_locked.count
+            # The current job has already been locked and will appear in the previous query
+            raise GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError if perform_concurrency > limit
+          end
+        end
+      end
+
+      class_methods do
+        def good_job_control_concurrency_with(config)
+          self.good_job_concurrency_config = config
+        end
+      end
+
+      def good_job_concurrency_key
+        key = self.class.good_job_concurrency_config[:key]
+        return if key.blank?
+
+        if key.respond_to? :call
+          instance_exec(&key)
+        else
+          key
+        end
+      end
+    end
+  end
+end

data/lib/good_job/current_execution.rb

@@ -4,11 +4,11 @@ module GoodJob
   # Thread-local attributes for passing values from Instrumentation.
   # (Cannot use ActiveSupport::CurrentAttributes because ActiveJob resets it)
   module CurrentExecution
-    # @!attribute [rw] error_on_retry
+    # @!attribute [rw] active_job_id
     #   @!scope class
-    #   Error captured by retry_on
-    #   @return [Exception, nil]
-    thread_mattr_accessor :error_on_retry
+    #   ActiveJob ID
+    #   @return [String, nil]
+    thread_mattr_accessor :active_job_id
 
     # @!attribute [rw] error_on_discard
     #   @!scope class
@@ -16,11 +16,18 @@ module GoodJob
     #   @return [Exception, nil]
     thread_mattr_accessor :error_on_discard
 
+    # @!attribute [rw] error_on_retry
+    #   @!scope class
+    #   Error captured by retry_on
+    #   @return [Exception, nil]
+    thread_mattr_accessor :error_on_retry
+
     # Resets attributes
     # @return [void]
     def self.reset
-      self.error_on_retry = nil
+      self.active_job_id = nil
       self.error_on_discard = nil
+      self.error_on_retry = nil
     end
 
     # @return [Integer] Current process ID

data/lib/good_job/job.rb

@@ -15,6 +15,7 @@ module GoodJob
     DEFAULT_PRIORITY = 0
 
     self.table_name = 'good_jobs'.freeze
+    self.advisory_lockable_column = 'id'.freeze
 
     attr_readonly :serialized_params
 
@@ -219,6 +220,21 @@ module GoodJob
           DEPRECATION
         end
 
+        if column_names.include?('concurrency_key')
+          good_job_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
+        else
+          ActiveSupport::Deprecation.warn(<<~DEPRECATION)
+            GoodJob has pending database migrations. To create the migration files, run:
+
+                rails generate good_job:update
+
+            To apply the migration files, run:
+
+                rails db:migrate
+
+          DEPRECATION
+        end
+
         good_job = GoodJob::Job.new(**good_job_args)
 
         instrument_payload[:good_job] = good_job
@@ -264,6 +280,10 @@ module GoodJob
       self.class.unscoped.unfinished.owns_advisory_locked.exists?(id: id)
     end
 
+    def active_job_id
+      super || serialized_params['job_id']
+    end
+
     private
 
     # @return [ExecutionResult]
@@ -273,6 +293,7 @@ module GoodJob
       )
 
       GoodJob::CurrentExecution.reset
+      GoodJob::CurrentExecution.active_job_id = active_job_id
       ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self, process_id: GoodJob::CurrentExecution.process_id, thread_name: GoodJob::CurrentExecution.thread_name }) do
         value = ActiveJob::Base.execute(params)
 

data/lib/good_job/lockable.rb

@@ -23,20 +23,20 @@ module GoodJob
 
     included do
       # Default column to be used when creating Advisory Locks
-      cattr_accessor(:advisory_lockable_column, instance_accessor: false) { primary_key }
+      class_attribute :advisory_lockable_column, instance_accessor: false, default: Concurrent::Delay.new { primary_key }
 
       # Default Postgres function to be used for Advisory Locks
-      cattr_accessor(:advisory_lockable_function) { "pg_try_advisory_lock" }
+      class_attribute :advisory_lockable_function, default: "pg_try_advisory_lock"
 
       # Attempt to acquire an advisory lock on the selected records and
       # return only those records for which a lock could be acquired.
-      # @!method advisory_lock(column: advisory_lockable_column, function: advisory_lockable_function)
+      # @!method advisory_lock(column: _advisory_lockable_column, function: advisory_lockable_function)
       # @!scope class
       # @param column [String, Symbol] column values to Advisory Lock against
       # @param function [String, Symbol]  Postgres Advisory Lock function name to use
       # @return [ActiveRecord::Relation]
       #   A relation selecting only the records that were locked.
-      scope :advisory_lock, (lambda do |column: advisory_lockable_column, function: advisory_lockable_function|
+      scope :advisory_lock, (lambda do |column: _advisory_lockable_column, function: advisory_lockable_function|
         original_query = self
 
         cte_table = Arel::Table.new(:rows)
@@ -48,15 +48,9 @@ module GoodJob
                    end
 
         composed_cte = Arel::Nodes::As.new(cte_table, Arel::Nodes::SqlLiteral.new([cte_type, "(", cte_query.to_sql, ")"].join(' ')))
-
-        # In addition to an advisory lock, there is also a FOR UPDATE SKIP LOCKED
-        # because this causes the query to skip jobs that were completed (and deleted)
-        # by another session in the time since the table snapshot was taken.
-        # In rare cases under high concurrency levels, leaving this out can result in double executions.
         query = cte_table.project(cte_table[:id])
                          .with(composed_cte)
                          .where(Arel.sql(sanitize_sql_for_conditions(["#{function}(('x' || substr(md5(:table_name || #{connection.quote_table_name(cte_table.name)}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
-                         .lock(Arel.sql("FOR UPDATE SKIP LOCKED"))
 
         limit = original_query.arel.ast.limit
         query.limit = limit.value if limit.present?
@@ -70,13 +64,13 @@ module GoodJob
       #
       # For details on +pg_locks+, see
       # {https://www.postgresql.org/docs/current/view-pg-locks.html}.
-      # @!method joins_advisory_locks(column: advisory_lockable_column)
+      # @!method joins_advisory_locks(column: _advisory_lockable_column)
       # @!scope class
       # @param column [String, Symbol] column values to Advisory Lock against
       # @return [ActiveRecord::Relation]
       # @example Get the records that have a session awaiting a lock:
       #   MyLockableRecord.joins_advisory_locks.where("pg_locks.granted = ?", false)
-      scope :joins_advisory_locks, (lambda do |column: advisory_lockable_column|
+      scope :joins_advisory_locks, (lambda do |column: _advisory_lockable_column|
         join_sql = <<~SQL.squish
           LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
             AND pg_locks.objsubid = 1
@@ -88,26 +82,26 @@ module GoodJob
       end)
 
       # Find records that do not have an advisory lock on them.
-      # @!method advisory_unlocked(column: advisory_lockable_column)
+      # @!method advisory_unlocked(column: _advisory_lockable_column)
       # @!scope class
       # @param column [String, Symbol] column values to Advisory Lock against
       # @return [ActiveRecord::Relation]
-      scope :advisory_unlocked, ->(column: advisory_lockable_column) { joins_advisory_locks(column: column).where(pg_locks: { locktype: nil }) }
+      scope :advisory_unlocked, ->(column: _advisory_lockable_column) { joins_advisory_locks(column: column).where(pg_locks: { locktype: nil }) }
 
       # Find records that have an advisory lock on them.
-      # @!method advisory_locked(column: advisory_lockable_column)
+      # @!method advisory_locked(column: _advisory_lockable_column)
       # @!scope class
       # @param column [String, Symbol] column values to Advisory Lock against
       # @return [ActiveRecord::Relation]
-      scope :advisory_locked, ->(column: advisory_lockable_column) { joins_advisory_locks(column: column).where.not(pg_locks: { locktype: nil }) }
+      scope :advisory_locked, ->(column: _advisory_lockable_column) { joins_advisory_locks(column: column).where.not(pg_locks: { locktype: nil }) }
 
       # Find records with advisory locks owned by the current Postgres
       # session/connection.
-      # @!method advisory_locked(column: advisory_lockable_column)
+      # @!method advisory_locked(column: _advisory_lockable_column)
       # @!scope class
       # @param column [String, Symbol] column values to Advisory Lock against
       # @return [ActiveRecord::Relation]
-      scope :owns_advisory_locked, ->(column: advisory_lockable_column) { joins_advisory_locks(column: column).where('"pg_locks"."pid" = pg_backend_pid()') }
+      scope :owns_advisory_locked, ->(column: _advisory_lockable_column) { joins_advisory_locks(column: column).where('"pg_locks"."pid" = pg_backend_pid()') }
 
       # Whether an advisory lock should be acquired in the same transaction
       # that created the record.
@@ -149,7 +143,7 @@ module GoodJob
       #   MyLockableRecord.order(created_at: :asc).limit(2).with_advisory_lock do |record|
       #     do_something_with record
       #   end
-      def with_advisory_lock(column: advisory_lockable_column, function: advisory_lockable_function, unlock_session: false)
+      def with_advisory_lock(column: _advisory_lockable_column, function: advisory_lockable_function, unlock_session: false)
         raise ArgumentError, "Must provide a block" unless block_given?
 
         records = advisory_lock(column: column, function: function).to_a
@@ -160,13 +154,19 @@ module GoodJob
             advisory_unlock_session
           else
             records.each do |record|
-              key = [table_name, record[advisory_lockable_column]].join
+              key = [table_name, record[_advisory_lockable_column]].join
               record.advisory_unlock(key: key, function: advisory_unlockable_function(function))
             end
           end
         end
       end
 
+      # Allow advisory_lockable_column to be a `Concurrent::Delay`
+      def _advisory_lockable_column
+        column = advisory_lockable_column
+        column.respond_to?(:value) ? column.value : column
+      end
+
       def supports_cte_materialization_specifiers?
         return @_supports_cte_materialization_specifiers if defined?(@_supports_cte_materialization_specifiers)
 
@@ -207,9 +207,16 @@ module GoodJob
     # @param function [String, Symbol] Postgres Advisory Lock function name to use
     # @return [Boolean] whether the lock was acquired.
     def advisory_lock(key: lockable_key, function: advisory_lockable_function)
-      query = <<~SQL.squish
-        SELECT #{function}(('x'||substr(md5($1::text), 1, 16))::bit(64)::bigint) AS locked
-      SQL
+      query = if function.include? "_try_"
+                <<~SQL.squish
+                  SELECT #{function}(('x'||substr(md5($1::text), 1, 16))::bit(64)::bigint) AS locked
+                SQL
+              else
+                <<~SQL.squish
+                  SELECT #{function}(('x'||substr(md5($1::text), 1, 16))::bit(64)::bigint)::text AS locked
+                SQL
+              end
+
       binds = [[nil, key]]
       self.class.connection.exec_query(pg_or_jdbc_query(query), 'GoodJob::Lockable Advisory Lock', binds).first['locked']
     end
@@ -307,7 +314,7 @@ module GoodJob
     # Default Advisory Lock key
     # @return [String]
     def lockable_key
-      [self.class.table_name, self[self.class.advisory_lockable_column]].join
+      [self.class.table_name, self[self.class._advisory_lockable_column]].join
     end
 
     delegate :pg_or_jdbc_query, to: :class

data/lib/good_job/notifier.rb

@@ -24,6 +24,8 @@ module GoodJob # :nodoc:
       max_queue: 1,
       fallback_policy: :discard,
     }.freeze
+    # Seconds to wait if database cannot be connected to
+    RECONNECT_INTERVAL = 5
     # Seconds to block while LISTENing for a message
     WAIT_INTERVAL = 1
 
@@ -114,7 +116,13 @@ module GoodJob # :nodoc:
         ActiveSupport::Notifications.instrument("notifier_notify_error.good_job", { error: thread_error })
       end
 
-      listen unless shutdown?
+      return if shutdown?
+
+      if thread_error.is_a?(ActiveRecord::ConnectionNotEstablished) || thread_error.is_a?(ActiveRecord::StatementInvalid)
+        listen(delay: RECONNECT_INTERVAL)
+      else
+        listen
+      end
     end
 
     private
@@ -125,8 +133,8 @@ module GoodJob # :nodoc:
       @executor = Concurrent::ThreadPoolExecutor.new(EXECUTOR_OPTIONS)
     end
 
-    def listen
-      future = Concurrent::Future.new(args: [@recipients, executor, @listening], executor: @executor) do |thr_recipients, thr_executor, thr_listening|
+    def listen(delay: 0)
+      future = Concurrent::ScheduledTask.new(delay, args: [@recipients, executor, @listening], executor: @executor) do |thr_recipients, thr_executor, thr_listening|
         with_listen_connection do |conn|
           ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
             conn.async_exec("LISTEN #{CHANNEL}").clear

data/lib/good_job/version.rb

@@ -1,4 +1,4 @@
 module GoodJob
   # GoodJob gem version.
-  VERSION = '1.10.0'.freeze
+  VERSION = '1.11.2'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.10.0
+  version: 1.11.2
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-06-29 00:00:00.000000000 Z
+date: 2021-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -362,6 +362,8 @@ files:
 - lib/generators/good_job/templates/update/migrations/03_add_active_job_id_index_and_concurrency_key_index_to_good_jobs.rb
 - lib/generators/good_job/update_generator.rb
 - lib/good_job.rb
+- lib/good_job/active_job_extensions.rb
+- lib/good_job/active_job_extensions/concurrency.rb
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
 - lib/good_job/configuration.rb