good_job 1.1.2 → 1.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4117e28d2a899ebb8b66a17fa79ff8495b36521d92f9146059c7b78406a361b6
-  data.tar.gz: 535dd8b9204cc76ee6749224217e5d33e23adc2414e8ca7c420d4deb35451efd
+  metadata.gz: b5125c48d26036d60649d0448f064aebcbe95fd90ac5e2eeb22e504761f7cdd1
+  data.tar.gz: f6cd3d0bcbfbbc639d0724557c7dbad61e9aef717545916b57a11c4413dcfe3f
 SHA512:
-  metadata.gz: 1f8bf7bf0f21604eb08b814d032b4a964f220b6d6384b368de7263672ece002f8fd01e5e2d77b0c6ed93e31ac03f235a2c40b8ad3f532de3fb3c755cd629ad24
-  data.tar.gz: 6006611999250e6b7708f381302379f1a07ba3522151630b708388436c9264ea2e2b4277c5753027171e82e7dd8a82b6f272386ad39170a192b94d0b36e45111
+  metadata.gz: eeb551e271b3a47aa903d64e566a010c87a632fd064835928d4986342405f458e02c5f88451c034ee2c73225274d54583ed963431e08e80232bcd3d87f76d500
+  data.tar.gz: f0c70c47ad3279bb41440ed27408511bdebc064524cf0933db066a0420bae829b1145b7713bc644febe0bc21e3010334f1c641863674110df878dd4245d17b7a

data/CHANGELOG.md

@@ -1,5 +1,84 @@
 # Changelog
 
+## [v1.2.2](https://github.com/bensheldon/good_job/tree/v1.2.2) (2020-08-26)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.1...v1.2.2)
+
+**Implemented enhancements:**
+
+- Run Github Action tests against Ruby 2.5, 2.6, 2.7 [\#100](https://github.com/bensheldon/good_job/issues/100)
+
+**Fixed bugs:**
+
+- Freezes puma on code change [\#95](https://github.com/bensheldon/good_job/issues/95)
+- Ruby 2.7 keyword arguments warning [\#93](https://github.com/bensheldon/good_job/issues/93)
+
+**Closed issues:**
+
+- Add test for `rails g good\_job:install` [\#57](https://github.com/bensheldon/good_job/issues/57)
+
+**Merged pull requests:**
+
+- Use more ActiveRecord in Lockable and not connection.execute [\#102](https://github.com/bensheldon/good_job/pull/102) ([bensheldon](https://github.com/bensheldon))
+- Run CI tests on Ruby 2.5, 2.6, and 2.7 [\#101](https://github.com/bensheldon/good_job/pull/101) ([arku](https://github.com/arku))
+- Return to using executor.wrap around Scheduler execution task [\#99](https://github.com/bensheldon/good_job/pull/99) ([bensheldon](https://github.com/bensheldon))
+- Fix Ruby 2.7 keyword arguments warning [\#98](https://github.com/bensheldon/good_job/pull/98) ([arku](https://github.com/arku))
+- Remove executor/reloader for less interlocking [\#97](https://github.com/bensheldon/good_job/pull/97) ([sj26](https://github.com/sj26))
+- Name the thread pools [\#96](https://github.com/bensheldon/good_job/pull/96) ([sj26](https://github.com/sj26))
+- Add test for `rails g good\_job:install` [\#94](https://github.com/bensheldon/good_job/pull/94) ([arku](https://github.com/arku))
+
+## [v1.2.1](https://github.com/bensheldon/good_job/tree/v1.2.1) (2020-08-21)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.0...v1.2.1)
+
+**Fixed bugs:**
+
+- undefined method `thread\_mattr\_accessor' when not requiring the Sprockets Railstie [\#85](https://github.com/bensheldon/good_job/issues/85)
+
+**Closed issues:**
+
+- Document comparison of GoodJob with other backends [\#51](https://github.com/bensheldon/good_job/issues/51)
+
+**Merged pull requests:**
+
+- Explicitly require thread\_mattr\_accessor from ActiveSupport [\#86](https://github.com/bensheldon/good_job/pull/86) ([bensheldon](https://github.com/bensheldon))
+- Add comparison of other backends to Readme [\#84](https://github.com/bensheldon/good_job/pull/84) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.2.0](https://github.com/bensheldon/good_job/tree/v1.2.0) (2020-08-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.4...v1.2.0)
+
+**Merged pull requests:**
+
+- Document GoodJob module [\#83](https://github.com/bensheldon/good_job/pull/83) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.4](https://github.com/bensheldon/good_job/tree/v1.1.4) (2020-08-19)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.3...v1.1.4)
+
+**Implemented enhancements:**
+
+- Explicitly name threads for easier debugging [\#64](https://github.com/bensheldon/good_job/issues/64)
+- Investigate Listen/Notify as alternative to polling [\#54](https://github.com/bensheldon/good_job/issues/54)
+
+**Merged pull requests:**
+
+- Add Postgres LISTEN/NOTIFY support [\#82](https://github.com/bensheldon/good_job/pull/82) ([bensheldon](https://github.com/bensheldon))
+- Allow Schedulers to filter \#create\_thread to avoid flood of queries when running async with multiple schedulers [\#81](https://github.com/bensheldon/good_job/pull/81) ([bensheldon](https://github.com/bensheldon))
+- Fully name scheduler threadpools and thread names; refactor CLI STDOUT [\#80](https://github.com/bensheldon/good_job/pull/80) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.3](https://github.com/bensheldon/good_job/tree/v1.1.3) (2020-08-14)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.2...v1.1.3)
+
+**Fixed bugs:**
+
+- Job exceptions not properly attached to good\_jobs record  [\#72](https://github.com/bensheldon/good_job/issues/72)
+
+**Merged pull requests:**
+
+- Document GoodJob::Scheduler with Yard [\#78](https://github.com/bensheldon/good_job/pull/78) ([bensheldon](https://github.com/bensheldon))
+
 ## [v1.1.2](https://github.com/bensheldon/good_job/tree/v1.1.2) (2020-08-13)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.1...v1.1.2)
@@ -22,6 +101,7 @@
 
 **Merged pull requests:**
 
+- Capture errors via instrumentation from retry\_on and discard\_on [\#79](https://github.com/bensheldon/good_job/pull/79) ([bensheldon](https://github.com/bensheldon))
 - Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
 - Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
 

data/README.md

@@ -6,11 +6,24 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 
 - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration. 
 - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby). 
-- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
+- **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
 For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
 
+<details>
+<summary><strong>📊 Comparison of GoodJob with other job queue backends (click to expand)</strong></summary>
+
+|                 | Queues, priority, retries | Database                              | Concurrency       | Reliability/Integrity  | Latency                  |
+|-----------------|---------------------------|---------------------------------------|-------------------|------------------------|--------------------------|
+| **GoodJob**     | ✅ Yes                     | ✅ Postgres                            | ✅ Multithreaded   | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
+| **Que**         | ✅ Yes                     | 🟨 Postgres, requires  `structure.sql` | ✅ Multithreaded   | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
+| **Delayed Job** | ✅ Yes                     | ✅ Postgres                            | 🟥 Single-threaded | ✅ ACID, record-based   | 🟨 Polling                |
+| **Sidekiq**     | ✅ Yes                     | 🟥 Redis                               | ✅ Multithreaded   | 🟥 Crashes lose jobs    | ✅ Redis BRPOP            |
+| **Sidekiq Pro** | ✅ Yes                     | 🟥 Redis                               | ✅ Multithreaded   | ✅ Redis RPOPLPUSH      | ✅ Redis RPOPLPUSH        |
+
+</details>
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -284,15 +297,20 @@ Depending on your application configuration, you may need to take additional ste
     # config/puma.rb
   
     before_fork do
-      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+      GoodJob.shutdown
     end
     
     on_worker_boot do
-      GoodJob::Scheduler.instances.each { |s| s.restart }
+      GoodJob.restart
     end
     
     on_worker_shutdown do
-      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+      GoodJob.shutdown
+    end
+  
+    MAIN_PID = Process.pid
+    at_exit do
+      GoodJob.shutdown if Process.pid == MAIN_PID
     end
     ```
   

data/exe/good_job

@@ -1,3 +1,4 @@
 #!/usr/bin/env ruby
 require 'good_job/cli'
+GOOD_JOB_LOG_TO_STDOUT = true
 GoodJob::CLI.start(ARGV)

data/lib/good_job.rb

@@ -10,14 +10,64 @@ require 'good_job/multi_scheduler'
 require 'good_job/adapter'
 require 'good_job/pg_locks'
 require 'good_job/performer'
+require 'good_job/current_execution'
+require 'good_job/notifier'
 
 require 'active_job/queue_adapters/good_job_adapter'
 
+# GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
+#
+# +GoodJob+ is the top-level namespace and exposes configuration attributes.
 module GoodJob
-  cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+  # @!attribute [rw] logger
+  #   @!scope class
+  #   The logger used by GoodJob
+  #   @return [Logger]
+  mattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+
+  # @!attribute [rw] preserve_job_records
+  #   @!scope class
+  #   Whether to preserve job records in the database after they have finished for inspection
+  #   @return [Boolean]
   mattr_accessor :preserve_job_records, default: false
+
+  # @!attribute [rw] reperform_jobs_on_standard_error
+  #   @!scope class
+  #   Whether to re-perform a job when a type of +StandardError+ is raised and bubbles up to the GoodJob backend
+  #   @return [Boolean]
   mattr_accessor :reperform_jobs_on_standard_error, default: true
+
+  # @!attribute [rw] on_thread_error
+  #   @!scope class
+  #   Called when a thread raises an error
+  #   @example Send errors to Sentry
+  #     # config/initializers/good_job.rb
+  #
+  #     # With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+  #     GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+  #   @return [#call, nil]
   mattr_accessor :on_thread_error, default: nil
 
+  # Shuts down all execution pools
+  # @param wait [Boolean] whether to wait for shutdown
+  # @return [void]
+  def self.shutdown(wait: true)
+    Notifier.instances.each { |adapter| adapter.shutdown(wait: wait) }
+    Scheduler.instances.each { |scheduler| scheduler.shutdown(wait: wait) }
+  end
+
+  # Tests if execution pools are shut down
+  # @return [Boolean] whether execution pools are shut down
+  def self.shutdown?
+    Notifier.instances.all?(&:shutdown?) && Scheduler.instances.all?(&:shutdown?)
+  end
+
+  # Restarts all execution pools
+  # @return [void]
+  def self.restart
+    Notifier.instances.each(&:restart)
+    Scheduler.instances.each(&:restart)
+  end
+
   ActiveSupport.run_load_hooks(:good_job, self)
 end

data/lib/good_job/adapter.rb

@@ -2,25 +2,29 @@ module GoodJob
   class Adapter
     EXECUTION_MODES = [:async, :external, :inline].freeze
 
-    def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
+    def initialize(execution_mode: nil, queues: nil, max_threads: nil, poll_interval: nil, scheduler: nil, notifier: nil, inline: false)
       if inline && execution_mode.nil?
         ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
         execution_mode = :inline
       end
 
-      configuration = GoodJob::Configuration.new({
-                                                   execution_mode: execution_mode,
-                                                   max_threads: max_threads,
-                                                   poll_interval: poll_interval,
-                                                 },
-                                                 env: ENV)
-
-      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(configuration.execution_mode)
+      configuration = GoodJob::Configuration.new(
+        {
+          execution_mode: execution_mode,
+          queues: queues,
+          max_threads: max_threads,
+          poll_interval: poll_interval,
+        }
+      )
 
       @execution_mode = configuration.execution_mode
+      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(@execution_mode)
 
-      @scheduler = scheduler
-      @scheduler = GoodJob::Scheduler.from_configuration(configuration) if @execution_mode == :async && @scheduler.blank?
+      if @execution_mode == :async # rubocop:disable Style/GuardClause
+        @notifier = notifier || GoodJob::Notifier.new
+        @scheduler = scheduler || GoodJob::Scheduler.from_configuration(configuration)
+        @notifier.recipients << [@scheduler, :create_thread]
+      end
     end
 
     def enqueue(active_job)
@@ -42,12 +46,14 @@ module GoodJob
         end
       end
 
-      @scheduler.create_thread if execute_async?
+      executed_locally = execute_async? && @scheduler.create_thread(queue_name: good_job.queue_name)
+      Notifier.notify(queue_name: good_job.queue_name) unless executed_locally
 
       good_job
     end
 
     def shutdown(wait: true)
+      @notifier&.shutdown(wait: wait)
       @scheduler&.shutdown(wait: wait)
     end
 

data/lib/good_job/cli.rb

@@ -18,8 +18,10 @@ module GoodJob
     def start
       set_up_application!
 
-      configuration = Configuration.new(options, env: ENV)
-      scheduler = Scheduler.from_configuration(configuration)
+      notifier = GoodJob::Notifier.new
+      configuration = GoodJob::Configuration.new(options)
+      scheduler = GoodJob::Scheduler.from_configuration(configuration)
+      notifier.recipients << [scheduler, :create_thread]
 
       @stop_good_job_executable = false
       %w[INT TERM].each do |signal|
@@ -28,9 +30,10 @@ module GoodJob
 
       Kernel.loop do
         sleep 0.1
-        break if @stop_good_job_executable || scheduler.shutdown?
+        break if @stop_good_job_executable || scheduler.shutdown? || notifier.shutdown?
       end
 
+      notifier.shutdown
       scheduler.shutdown
     end
 
@@ -41,6 +44,7 @@ module GoodJob
                   type: :numeric,
                   default: 24 * 60 * 60,
                   desc: "Delete records finished more than this many seconds ago"
+
     def cleanup_preserved_jobs
       set_up_application!
 
@@ -55,6 +59,10 @@ module GoodJob
     no_commands do
       def set_up_application!
         require RAILS_ENVIRONMENT_RB
+        return unless defined?(GOOD_JOB_LOG_TO_STDOUT) && GOOD_JOB_LOG_TO_STDOUT && !ActiveSupport::Logger.logger_outputs_to?(GoodJob.logger, STDOUT)
+
+        GoodJob::LogSubscriber.loggers << ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+        GoodJob::LogSubscriber.reset_logger
       end
     end
   end

data/lib/good_job/current_execution.rb

@@ -0,0 +1,37 @@
+require 'active_support/core_ext/module/attribute_accessors_per_thread'
+
+module GoodJob
+  # Thread-local attributes for passing values from Instrumentation.
+  # (Cannot use ActiveSupport::CurrentAttributes because ActiveJob resets it)
+
+  module CurrentExecution
+    # @!attribute [rw] error_on_retry
+    #   @!scope class
+    #   Error captured by retry_on
+    #   @return [Exception, nil]
+    thread_mattr_accessor :error_on_retry
+
+    # @!attribute [rw] error_on_discard
+    #   @!scope class
+    #   Error captured by discard_on
+    #   @return [Exception, nil]
+    thread_mattr_accessor :error_on_discard
+
+    # Resets attributes
+    # @return [void]
+    def self.reset
+      self.error_on_retry = nil
+      self.error_on_discard = nil
+    end
+
+    # @return [Integer] Current process ID
+    def self.process_id
+      Process.pid
+    end
+
+    # @return [String] Current thread name
+    def self.thread_name
+      (Thread.current.name || Thread.current.object_id).to_s
+    end
+  end
+end

data/lib/good_job/job.rb

@@ -9,6 +9,25 @@ module GoodJob
 
     self.table_name = 'good_jobs'.freeze
 
+    def self.queue_parser(string)
+      string = string.presence || '*'
+
+      if string.first == '-'
+        exclude_queues = true
+        string = string[1..-1]
+      end
+
+      queues = string.split(',').map(&:strip)
+
+      if queues.include?('*')
+        { all: true }
+      elsif exclude_queues
+        { exclude: queues }
+      else
+        { include: queues }
+      end
+    end
+
     scope :unfinished, (lambda do
       if column_names.include?('finished_at')
         where(finished_at: nil)
@@ -21,20 +40,14 @@ module GoodJob
     scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
     scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
     scope :queue_string, (lambda do |string|
-      string = string.presence || '*'
-
-      if string.first == '-'
-        exclude_queues = true
-        string = string[1..-1]
-      end
-
-      queue_names_without_all = string.split(',').map(&:strip).reject { |q| q == '*' }
-      return if queue_names_without_all.size.zero?
-
-      if exclude_queues
-        where.not(queue_name: queue_names_without_all).or where(queue_name: nil)
-      else
-        where(queue_name: queue_names_without_all)
+      parsed = queue_parser(string)
+
+      if parsed[:all]
+        all
+      elsif parsed[:exclude]
+        where.not(queue_name: parsed[:exclude]).or where(queue_name: nil)
+      elsif parsed[:include]
+        where(queue_name: parsed[:include])
       end
     end)
 
@@ -45,7 +58,8 @@ module GoodJob
 
       unfinished.priority_ordered.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
         good_job = good_jobs.first
-        break unless good_job
+        # TODO: Determine why some records are fetched without an advisory lock at all
+        break unless good_job&.owns_advisory_lock?
 
         result, error = good_job.perform
       end
@@ -60,7 +74,7 @@ module GoodJob
           queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
           priority: active_job.priority || DEFAULT_PRIORITY,
           serialized_params: active_job.serialize,
-          scheduled_at: scheduled_at || Time.current,
+          scheduled_at: scheduled_at,
           create_with_advisory_lock: create_with_advisory_lock
         )
 
@@ -76,11 +90,11 @@ module GoodJob
     def perform(destroy_after: !GoodJob.preserve_job_records, reperform_on_standard_error: GoodJob.reperform_jobs_on_standard_error)
       raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
 
+      GoodJob::CurrentExecution.reset
       result = nil
       rescued_error = nil
       error = nil
 
-      ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
       self.performed_at = Time.current
       save! unless destroy_after
 
@@ -89,18 +103,23 @@ module GoodJob
       )
 
       begin
-        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
+        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self, process_id: GoodJob::CurrentExecution.process_id, thread_name: GoodJob::CurrentExecution.thread_name }) do
           result = ActiveJob::Base.execute(params)
         end
       rescue StandardError => e
         rescued_error = e
       end
 
+      retry_or_discard_error = GoodJob::CurrentExecution.error_on_retry ||
+                               GoodJob::CurrentExecution.error_on_discard
+
       if rescued_error
         error = rescued_error
       elsif result.is_a?(Exception)
         error = result
         result = nil
+      elsif retry_or_discard_error
+        error = retry_or_discard_error
       end
 
       error_message = "#{error.class}: #{error.message}" if error

data/lib/good_job/lockable.rb

@@ -55,19 +55,17 @@ module GoodJob
     end
 
     def advisory_lock
-      query = <<~SQL
-        SELECT 1 AS one
-        WHERE pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
+      where_sql = <<~SQL
+        pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
       SQL
-      self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
+      self.class.unscoped.where(where_sql, { table_name: self.class.table_name, id: send(self.class.primary_key) }).exists?
     end
 
     def advisory_unlock
-      query = <<~SQL
-        SELECT 1 AS one
-        WHERE pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
+      where_sql = <<~SQL
+        pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
       SQL
-      self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
+      self.class.unscoped.where(where_sql, { table_name: self.class.table_name, id: send(self.class.primary_key) }).exists?
     end
 
     def advisory_lock!
@@ -85,11 +83,11 @@ module GoodJob
     end
 
     def advisory_locked?
-      self.class.advisory_locked.where(id: send(self.class.primary_key)).any?
+      self.class.unscoped.advisory_locked.where(id: send(self.class.primary_key)).exists?
     end
 
     def owns_advisory_lock?
-      self.class.owns_advisory_locked.where(id: send(self.class.primary_key)).any?
+      self.class.unscoped.owns_advisory_locked.where(id: send(self.class.primary_key)).exists?
     end
 
     def advisory_unlock!

data/lib/good_job/log_subscriber.rb

@@ -32,7 +32,7 @@ module GoodJob
       performer_name = event.payload[:performer_name]
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob started scheduler with queues=#{performer_name} max_threads=#{max_threads} poll_interval=#{poll_interval}."
       end
     end
@@ -40,7 +40,7 @@ module GoodJob
     def scheduler_shutdown_start(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob shutting down scheduler..."
       end
     end
@@ -48,7 +48,7 @@ module GoodJob
     def scheduler_shutdown(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob scheduler is shut down."
       end
     end
@@ -56,24 +56,100 @@ module GoodJob
     def scheduler_restart_pools(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob scheduler has restarted."
       end
     end
 
+    def perform_job(event)
+      good_job = event.payload[:good_job]
+      process_id = event.payload[:process_id]
+      thread_name = event.payload[:thread_name]
+
+      info(tags: [process_id, thread_name]) do
+        "Executed GoodJob #{good_job.id}"
+      end
+    end
+
+    def notifier_listen(_event)
+      info do
+        "Notifier subscribed with LISTEN"
+      end
+    end
+
+    def notifier_notified(event)
+      payload = event.payload[:payload]
+
+      debug do
+        "Notifier received payload: #{payload}"
+      end
+    end
+
+    def notifier_notify_error(event)
+      error = event.payload[:error]
+
+      error do
+        "Notifier errored: #{error}"
+      end
+    end
+
+    def notifier_unlisten(_event)
+      info do
+        "Notifier unsubscribed with UNLISTEN"
+      end
+    end
+
     def cleanup_preserved_jobs(event)
       timestamp = event.payload[:timestamp]
       deleted_records_count = event.payload[:deleted_records_count]
 
-      info_and_stdout do
+      info do
         "GoodJob deleted #{deleted_records_count} preserved #{'job'.pluralize(deleted_records_count)} finished before #{timestamp}."
       end
     end
 
-    private
+    class << self
+      def loggers
+        @_loggers ||= [GoodJob.logger]
+      end
+
+      def logger
+        @_logger ||= begin
+                      logger = Logger.new(StringIO.new)
+                      loggers.each do |each_logger|
+                        logger.extend(ActiveSupport::Logger.broadcast(each_logger))
+                      end
+                      logger
+                    end
+      end
+
+      def reset_logger
+        @_logger = nil
+      end
+    end
 
     def logger
-      GoodJob.logger
+      GoodJob::LogSubscriber.logger
+    end
+
+    private
+
+    def tag_logger(*tags, &block)
+      tags = tags.dup.unshift("GoodJob").compact
+
+      self.class.loggers.inject(block) do |inner, each_logger|
+        if each_logger.respond_to?(:tagged)
+          tags_for_logger = if each_logger.formatter.current_tags.include?("ActiveJob")
+                              ["ActiveJob"] + tags
+                            else
+                              tags
+                            end
+
+          proc { each_logger.tagged(*tags_for_logger, &inner) }
+        else
+          inner
+        end
+      end.call
     end
 
     %w(info debug warn error fatal unknown).each do |level|
@@ -81,30 +157,11 @@ module GoodJob
         def #{level}(progname = nil, tags: [], &block)
           return unless logger
 
-          if logger.respond_to?(:tagged)
-            tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
-            logger.tagged(*tags.compact) do
-              logger.#{level}(progname, &block)
-            end
-          else
+          tag_logger(*tags) do
             logger.#{level}(progname, &block)
           end
         end
       METHOD
     end
-
-    def info_and_stdout(progname = nil, tags: [], &block)
-      unless ActiveSupport::Logger.logger_outputs_to?(logger, STDOUT)
-        tags_string = (['GoodJob'] + tags).map { |t| "[#{t}]" }.join(' ')
-        stdout_message = "#{tags_string} #{yield}"
-        $stdout.puts stdout_message
-      end
-
-      info(progname, tags: [], &block)
-    end
-
-    def thread_name
-      Thread.current.name || Thread.current.object_id
-    end
   end
 end

data/lib/good_job/multi_scheduler.rb

@@ -18,8 +18,17 @@ module GoodJob
       schedulers.each { |s| s.restart(wait: wait) }
     end
 
-    def create_thread
-      schedulers.all?(&:create_thread)
+    def create_thread(state = nil)
+      results = []
+      any_true = schedulers.any? do |scheduler|
+        scheduler.create_thread(state).tap { |result| results << result }
+      end
+
+      if any_true
+        true
+      else
+        results.any? { |result| result == false } ? false : nil
+      end
     end
   end
 end

data/lib/good_job/notifier.rb

@@ -0,0 +1,127 @@
+require 'concurrent/atomic/atomic_boolean'
+
+module GoodJob # :nodoc:
+  #
+  # Wrapper for Postgres LISTEN/NOTIFY
+  #
+  class Notifier
+    CHANNEL = 'good_job'.freeze
+    POOL_OPTIONS = {
+      name: name,
+      min_threads: 0,
+      max_threads: 1,
+      auto_terminate: true,
+      idletime: 60,
+      max_queue: 1,
+      fallback_policy: :discard,
+    }.freeze
+    WAIT_INTERVAL = 1
+
+    # @!attribute [r] instances
+    #   @!scope class
+    #   @return [array<GoodJob:Adapter>] the instances of +GoodJob::Notifier+
+    cattr_reader :instances, default: [], instance_reader: false
+
+    def self.notify(message)
+      connection = ActiveRecord::Base.connection
+      connection.exec_query <<~SQL
+        NOTIFY #{CHANNEL}, #{connection.quote(message.to_json)}
+      SQL
+    end
+
+    attr_reader :recipients
+
+    def initialize(*recipients)
+      @recipients = Concurrent::Array.new(recipients)
+      @listening = Concurrent::AtomicBoolean.new(false)
+
+      self.class.instances << self
+
+      create_pool
+      listen
+    end
+
+    def listening?
+      @listening.true?
+    end
+
+    def restart(wait: true)
+      shutdown(wait: wait)
+      create_pool
+      listen
+    end
+
+    def shutdown(wait: true)
+      return unless @pool.running?
+
+      @pool.shutdown
+      @pool.wait_for_termination if wait
+    end
+
+    def shutdown?
+      !@pool.running?
+    end
+
+    private
+
+    def create_pool
+      @pool = Concurrent::ThreadPoolExecutor.new(POOL_OPTIONS)
+    end
+
+    def listen
+      future = Concurrent::Future.new(args: [@recipients, @pool, @listening], executor: @pool) do |recipients, pool, listening|
+        begin
+          with_listen_connection do |conn|
+            ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
+              conn.async_exec "LISTEN #{CHANNEL}"
+            end
+
+            ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
+              while pool.running?
+                listening.make_true
+                conn.wait_for_notify(WAIT_INTERVAL) do |channel, _pid, payload|
+                  listening.make_false
+                  next unless channel == CHANNEL
+
+                  ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
+                  parsed_payload = JSON.parse(payload, symbolize_names: true)
+                  recipients.each do |recipient|
+                    target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
+                    target.send(method_name, parsed_payload)
+                  end
+                end
+                listening.make_false
+              end
+            end
+          end
+        rescue StandardError => e
+          ActiveSupport::Notifications.instrument("notifier_notify_error.good_job", { error: e })
+          raise
+        ensure
+          @listening.make_false
+          ActiveSupport::Notifications.instrument("notifier_unlisten.good_job") do
+            conn.async_exec "UNLISTEN *"
+          end
+        end
+      end
+
+      future.add_observer(self, :listen_observer)
+      future.execute
+    end
+
+    def listen_observer(_time, _result, _thread_error)
+      listen unless shutdown?
+    end
+
+    def with_listen_connection
+      ar_conn = ActiveRecord::Base.connection_pool.checkout.tap do |conn|
+        ActiveRecord::Base.connection_pool.remove(conn)
+      end
+      pg_conn = ar_conn.raw_connection
+      pg_conn.exec("SET application_name = #{pg_conn.escape_identifier(self.class.name)}")
+      yield pg_conn
+    ensure
+      ar_conn.disconnect!
+    end
+  end
+end

data/lib/good_job/performer.rb

@@ -2,14 +2,21 @@ module GoodJob
   class Performer
     attr_reader :name
 
-    def initialize(target, method_name, name: nil)
+    def initialize(target, method_name, name: nil, filter: nil)
       @target = target
       @method_name = method_name
       @name = name
+      @filter = filter
     end
 
     def next
       @target.public_send(@method_name)
     end
+
+    def next?(state = {})
+      return true unless @filter.respond_to?(:call)
+
+      @filter.call(state)
+    end
   end
 end

data/lib/good_job/railtie.rb

@@ -4,5 +4,15 @@ module GoodJob
       ActiveSupport.on_load(:good_job) { self.logger = ::Rails.logger }
       GoodJob::LogSubscriber.attach_to :good_job
     end
+
+    initializer "good_job.active_job_notifications" do
+      ActiveSupport::Notifications.subscribe "enqueue_retry.active_job" do |event|
+        GoodJob::CurrentExecution.error_on_retry = event.payload[:error]
+      end
+
+      ActiveSupport::Notifications.subscribe "discard.active_job" do |event|
+        GoodJob::CurrentExecution.error_on_discard = event.payload[:error]
+      end
+    end
   end
 end

data/lib/good_job/scheduler.rb

@@ -2,16 +2,26 @@ require "concurrent/executor/thread_pool_executor"
 require "concurrent/timer_task"
 require "concurrent/utility/processor_counter"
 
-module GoodJob
+module GoodJob # :nodoc:
+  #
+  # Schedulers are generic thread execution pools that are responsible for
+  # periodically checking for available execution tasks, executing tasks in a
+  # bounded thread-pool, and efficiently scaling execution threads.
+  #
+  # Schedulers are "generic" in the sense that they delegate task execution
+  # details to a "Performer" object that responds to #next.
+  #
   class Scheduler
+    # Defaults for instance of Concurrent::TimerTask
     DEFAULT_TIMER_OPTIONS = {
       execution_interval: 1,
       timeout_interval: 1,
       run_now: true,
     }.freeze
 
+    # Defaults for instance of Concurrent::ThreadPoolExecutor
     DEFAULT_POOL_OPTIONS = {
-      name: 'good_job',
+      name: name,
       min_threads: 0,
       max_threads: Concurrent.processor_count,
       auto_terminate: true,
@@ -20,18 +30,35 @@ module GoodJob
       fallback_policy: :discard,
     }.freeze
 
+    # @!attribute [r] instances
+    #   @!scope class
+    #   All instantiated Schedulers in the current process.
+    #   @return [array<GoodJob:Scheduler>]
     cattr_reader :instances, default: [], instance_reader: false
 
+    # Creates GoodJob::Scheduler(s) and Performers from a GoodJob::Configuration instance.
+    # @param configuration [GoodJob::Configuration]
+    # @return [GoodJob::Scheduler, GoodJob::MultiScheduler]
     def self.from_configuration(configuration)
       schedulers = configuration.queue_string.split(';').map do |queue_string_and_max_threads|
         queue_string, max_threads = queue_string_and_max_threads.split(':')
         max_threads = (max_threads || configuration.max_threads).to_i
 
         job_query = GoodJob::Job.queue_string(queue_string)
-        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
+        parsed = GoodJob::Job.queue_parser(queue_string)
+        job_filter = proc do |state|
+          if parsed[:exclude]
+            !parsed[:exclude].include? state[:queue_name]
+          elsif parsed[:include]
+            parsed[:include].include? state[:queue_name]
+          else
+            true
+          end
+        end
+        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string, filter: job_filter)
 
         timer_options = {}
-        timer_options[:execution_interval] = configuration.poll_interval if configuration.poll_interval.positive?
+        timer_options[:execution_interval] = configuration.poll_interval
 
         pool_options = {
           max_threads: max_threads,
@@ -47,6 +74,9 @@ module GoodJob
       end
     end
 
+    # @param performer [GoodJob::Performer]
+    # @param timer_options [Hash] Options to instantiate a Concurrent::TimerTask
+    # @param pool_options [Hash] Options to instantiate a Concurrent::ThreadPoolExecutor
     def initialize(performer, timer_options: {}, pool_options: {})
       raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 
@@ -56,14 +86,19 @@ module GoodJob
       @pool_options = DEFAULT_POOL_OPTIONS.merge(pool_options)
       @timer_options = DEFAULT_TIMER_OPTIONS.merge(timer_options)
 
+      @pool_options[:name] = "GoodJob::Scheduler(queues=#{@performer.name} max_threads=#{@pool_options[:max_threads]} poll_interval=#{@timer_options[:execution_interval]})"
+
       create_pools
     end
 
+    # Shut down the Scheduler.
+    # @param wait [Boolean] Wait for actively executing jobs to finish
+    # @return [void]
     def shutdown(wait: true)
       @_shutdown = true
 
-      ActiveSupport::Notifications.instrument("scheduler_shutdown_start.good_job", { wait: wait, process_id: process_id })
-      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait, process_id: process_id }) do
+      instrument("scheduler_shutdown_start", { wait: wait })
+      instrument("scheduler_shutdown", { wait: wait }) do
         if @timer&.running?
           @timer.shutdown
           @timer.wait_for_termination if wait
@@ -76,19 +111,32 @@ module GoodJob
       end
     end
 
+    # True when the Scheduler is shutdown.
+    # @return [true, false, nil]
     def shutdown?
       @_shutdown
     end
 
+    # Restart the Scheduler. When shutdown, start; or shutdown and start.
+    # @param wait [Boolean] Wait for actively executing jobs to finish
+    # @return [void]
     def restart(wait: true)
-      ActiveSupport::Notifications.instrument("scheduler_restart_pools.good_job", { process_id: process_id }) do
+      instrument("scheduler_restart_pools") do
         shutdown(wait: wait) unless shutdown?
         create_pools
+        @_shutdown = false
       end
     end
 
-    def create_thread
-      return false unless @pool.ready_worker_count.positive?
+    # Triggers a Performer execution, if an execution thread is available.
+    # @param state [nil, Object] Allows Performer#next? to accept or reject the execution
+    # @return [nil, Boolean] if the thread was created
+    def create_thread(state = nil)
+      return nil unless @pool.running? && @pool.ready_worker_count.positive?
+
+      if state
+        return false unless @performer.next?(state)
+      end
 
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
         output = nil
@@ -97,35 +145,32 @@ module GoodJob
       end
       future.add_observer(self, :task_observer)
       future.execute
+
+      true
     end
 
+    # Invoked on completion of TimerTask task.
+    # @!visibility private
+    # @return [void]
     def timer_observer(time, executed_task, thread_error)
       GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
-      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
+      instrument("finished_timer_task", { result: executed_task, error: thread_error, time: time })
     end
 
+    # Invoked on completion of ThreadPoolExecutor task
+    # @!visibility private
+    # @return [void]
     def task_observer(time, output, thread_error)
       GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
-      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
+      instrument("finished_job_task", { result: output, error: thread_error, time: time })
       create_thread if output
     end
 
-    class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
-      # https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
-      def ready_worker_count
-        synchronize do
-          workers_still_to_be_created = @max_length - @pool.length
-          workers_created_but_waiting = @ready.length
-
-          workers_still_to_be_created + workers_created_but_waiting
-        end
-      end
-    end
-
     private
 
+    # @return [void]
     def create_pools
-      ActiveSupport::Notifications.instrument("scheduler_create_pools.good_job", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval], process_id: process_id }) do
+      instrument("scheduler_create_pools", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval] }) do
         @pool = ThreadPoolExecutor.new(@pool_options)
         next unless @timer_options[:execution_interval].positive?
 
@@ -135,12 +180,29 @@ module GoodJob
       end
     end
 
-    def process_id
-      Process.pid
+    def instrument(name, payload = {}, &block)
+      payload = payload.reverse_merge({
+                                        scheduler: self,
+                                        process_id: GoodJob::CurrentExecution.process_id,
+                                        thread_name: GoodJob::CurrentExecution.thread_name,
+                                      })
+
+      ActiveSupport::Notifications.instrument("#{name}.good_job", payload, &block)
     end
+  end
 
-    def thread_name
-      Thread.current.name || Thread.current.object_id
+  # Slightly customized sub-class of Concurrent::ThreadPoolExecutor
+  class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
+    # Number of idle or potential threads available to execute tasks
+    # https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
+    # @return [Integer]
+    def ready_worker_count
+      synchronize do
+        workers_still_to_be_created = @max_length - @pool.length
+        workers_created_but_waiting = @ready.length
+
+        workers_still_to_be_created + workers_created_but_waiting
+      end
     end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '1.1.2'.freeze
+  VERSION = '1.2.2'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.1.2
+  version: 1.2.2
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-08-13 00:00:00.000000000 Z
+date: 2020-08-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -151,7 +151,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: pry
+  name: pry-rails
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -178,6 +178,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rbtrace
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
@@ -248,6 +262,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: sigdump
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
 email:
 - bensheldon@gmail.com
@@ -270,10 +312,12 @@ files:
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
 - lib/good_job/configuration.rb
+- lib/good_job/current_execution.rb
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
 - lib/good_job/log_subscriber.rb
 - lib/good_job/multi_scheduler.rb
+- lib/good_job/notifier.rb
 - lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb