good_job 1.1.1 → 1.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b7b7100f183bea75e74273f8c60b87eb00f234b56196f1239ad5c1677e216250
-  data.tar.gz: b22120c6cdb91f38e201ef2fd8dd150f7775462ba27e7b9646bad504384c94d0
+  metadata.gz: '08edeaef5cda608f5e5afd0ee0c8813805e4e88791a338dc66ab9129a9f727f6'
+  data.tar.gz: 3608a46f95035a843ea422cca420f344d870a8be558fe9f2cdb6c765c5cbd153
 SHA512:
-  metadata.gz: '0844d72f1fb34e1608f40d6c134a3a24e8e5e0876ca4c96eaa246568db87f64b47be8170e5061517f455148da5e40fdd0a278a8a790a93a7ddb64eb024fdb35f'
-  data.tar.gz: 20934ee8ab6fc277519c2a551ee44d6803d09405215faff35fda4227e88729d720968235003e694f83887cea82709caa68eabc7afbb10335b2a671b3bddfe604
+  metadata.gz: 57f471ffef16a4f1def70922e8601b3612de84441233523492a543b7276dbb00c5dd8e37a8fd7c58d930ac78c3ea0f65704a4bef4175ef588fbf65d11051eca7
+  data.tar.gz: e0d76a4990f613b81431e42c89f5ff1172b5353a9025d6684d866048771cbd845d51a12a212fe1df85934d7854f298a333d426b8738b0fbfd2fbbad27880f02a

data/CHANGELOG.md

@@ -1,5 +1,67 @@
 # Changelog
 
+## [v1.2.1](https://github.com/bensheldon/good_job/tree/v1.2.1) (2020-08-21)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.2.0...v1.2.1)
+
+**Closed issues:**
+
+- undefined method `thread\_mattr\_accessor' when not requiring the Sprockets Railstie [\#85](https://github.com/bensheldon/good_job/issues/85)
+- Document comparison of GoodJob with other backends [\#51](https://github.com/bensheldon/good_job/issues/51)
+
+**Merged pull requests:**
+
+- Explicitly require thread\_mattr\_accessor from ActiveSupport [\#86](https://github.com/bensheldon/good_job/pull/86) ([bensheldon](https://github.com/bensheldon))
+- Add comparison of other backends to Readme [\#84](https://github.com/bensheldon/good_job/pull/84) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.2.0](https://github.com/bensheldon/good_job/tree/v1.2.0) (2020-08-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.4...v1.2.0)
+
+**Merged pull requests:**
+
+- Document GoodJob module [\#83](https://github.com/bensheldon/good_job/pull/83) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.4](https://github.com/bensheldon/good_job/tree/v1.1.4) (2020-08-19)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.3...v1.1.4)
+
+**Implemented enhancements:**
+
+- Explicitly name threads for easier debugging [\#64](https://github.com/bensheldon/good_job/issues/64)
+- Investigate Listen/Notify as alternative to polling [\#54](https://github.com/bensheldon/good_job/issues/54)
+
+**Merged pull requests:**
+
+- Add Postgres LISTEN/NOTIFY support [\#82](https://github.com/bensheldon/good_job/pull/82) ([bensheldon](https://github.com/bensheldon))
+- Allow Schedulers to filter \#create\_thread to avoid flood of queries when running async with multiple schedulers [\#81](https://github.com/bensheldon/good_job/pull/81) ([bensheldon](https://github.com/bensheldon))
+- Fully name scheduler threadpools and thread names; refactor CLI STDOUT [\#80](https://github.com/bensheldon/good_job/pull/80) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.3](https://github.com/bensheldon/good_job/tree/v1.1.3) (2020-08-14)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.2...v1.1.3)
+
+**Fixed bugs:**
+
+- Job exceptions not properly attached to good\_jobs record  [\#72](https://github.com/bensheldon/good_job/issues/72)
+
+**Merged pull requests:**
+
+- Capture errors via instrumentation from retry\_on and discard\_on [\#79](https://github.com/bensheldon/good_job/pull/79) ([bensheldon](https://github.com/bensheldon))
+- Document GoodJob::Scheduler with Yard [\#78](https://github.com/bensheldon/good_job/pull/78) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.2](https://github.com/bensheldon/good_job/tree/v1.1.2) (2020-08-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.1...v1.1.2)
+
+**Implemented enhancements:**
+
+- Allow the omission of queue names within a scheduler [\#73](https://github.com/bensheldon/good_job/issues/73)
+
+**Merged pull requests:**
+
+- Allow named queues to be excluded with a minus [\#77](https://github.com/bensheldon/good_job/pull/77) ([bensheldon](https://github.com/bensheldon))
+
 ## [v1.1.1](https://github.com/bensheldon/good_job/tree/v1.1.1) (2020-08-12)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.0...v1.1.1)
@@ -124,7 +186,6 @@
 - Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
 - Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
 - Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))
-- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
 
@@ -140,6 +201,7 @@
 - Improve generation of changelog [\#36](https://github.com/bensheldon/good_job/pull/36) ([bensheldon](https://github.com/bensheldon))
 - Update Github Action Workflow for Backlog Project Board [\#35](https://github.com/bensheldon/good_job/pull/35) ([bensheldon](https://github.com/bensheldon))
 - Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
+- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
 - Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)

data/README.md

@@ -6,11 +6,24 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 
 - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration. 
 - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby). 
-- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
+- **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
 For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
 
+<details>
+<summary><strong>📊 Comparison of GoodJob with other job queue backends (click to expand)</strong></summary>
+
+|                 | Queues, priority, retries | Database                              | Concurrency       | Reliability/Integrity  | Latency                  |
+|-----------------|---------------------------|---------------------------------------|-------------------|------------------------|--------------------------|
+| **GoodJob**     | ✅ Yes                     | ✅ Postgres                            | ✅ Multithreaded   | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
+| **Que**         | ✅ Yes                     | 🟨 Postgres, requires  `structure.sql` | ✅ Multithreaded   | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
+| **Delayed Job** | ✅ Yes                     | ✅ Postgres                            | 🟥 Single-threaded | ✅ ACID, record-based   | 🟨 Polling                |
+| **Sidekiq**     | ✅ Yes                     | 🟥 Redis                               | ✅ Multithreaded   | 🟥 Crashes lose jobs    | ✅ Redis BRPOP            |
+| **Sidekiq Pro** | ✅ Yes                     | 🟥 Redis                               | ✅ Multithreaded   | ✅ Redis RPOPLPUSH      | ✅ Redis RPOPLPUSH        |
+
+</details>
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -79,29 +92,36 @@ $ bundle install
       good_job start
     
     Options:
-      [--max-threads=N]                           # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
-      [--queues=queue1,queue2(;queue3,queue4:5)]  # Queues to work from. Separate multiple queues with commas; separate isolated execution pools with semicolons and threads with colons (default: *)
-      [--poll-interval=N]                         # Interval between polls for available jobs in seconds (default: 1)
+      [--max-threads=N]                                          # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
+      [--queues=queue1,queue2(;queue3,queue4:5;-queue1,queue2)]  # Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)
+      [--poll-interval=N]                                        # Interval between polls for available jobs in seconds (default: 1)    
     
     Start job worker
     ```
 
-1. Optimize execution to reduce congestion and execution latency. By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
+1. Optimize execution to reduce congestion and execution latency. 
+
+    By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
 
     - Multiple execution pools within a single process:
         
         ```bash
-        $ bundle exec good_job --queues=*;transactional_messages:2;batch_processing:1 --max-threads=5
+        $ bundle exec good_job --queues=transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;* --max-threads=5
         ```
       
-        This configuration will result in a single process with 3 isolated thread execution pools. A pool that will run jobs from any queue, `*`, with up to 5 threads; a pool that will only run jobs enqueued on `transactional_messages` with up to 2 threads; and a pool dedicated to the `batch_processing` queue with a single thread.
+        This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
+        
+        - `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
+        - `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
+        - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
+        - `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
         
         For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy. 
         
         Configuration can be injected by environment variables too:
         
         ```bash
-        $ GOOD_JOB_QUEUES="*;transactional_messages:2;batch_processing:1" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
+        $ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
         ```
         
     - Multiple processes; for example, on Heroku:
@@ -277,15 +297,20 @@ Depending on your application configuration, you may need to take additional ste
     # config/puma.rb
   
     before_fork do
-      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+      GoodJob.shutdown
     end
     
     on_worker_boot do
-      GoodJob::Scheduler.instances.each { |s| s.restart }
+      GoodJob.restart
     end
     
     on_worker_shutdown do
-      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+      GoodJob.shutdown
+    end
+  
+    MAIN_PID = Process.pid
+    at_exit do
+      GoodJob.shutdown if Process.pid == MAIN_PID
     end
     ```
   

data/exe/good_job

@@ -1,3 +1,4 @@
 #!/usr/bin/env ruby
 require 'good_job/cli'
+GOOD_JOB_LOG_TO_STDOUT = true
 GoodJob::CLI.start(ARGV)

data/lib/good_job.rb

@@ -10,14 +10,64 @@ require 'good_job/multi_scheduler'
 require 'good_job/adapter'
 require 'good_job/pg_locks'
 require 'good_job/performer'
+require 'good_job/current_execution'
+require 'good_job/notifier'
 
 require 'active_job/queue_adapters/good_job_adapter'
 
+# GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
+#
+# +GoodJob+ is the top-level namespace and exposes configuration attributes.
 module GoodJob
-  cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+  # @!attribute [rw] logger
+  #   @!scope class
+  #   The logger used by GoodJob
+  #   @return [Logger]
+  mattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+
+  # @!attribute [rw] preserve_job_records
+  #   @!scope class
+  #   Whether to preserve job records in the database after they have finished for inspection
+  #   @return [Boolean]
   mattr_accessor :preserve_job_records, default: false
+
+  # @!attribute [rw] reperform_jobs_on_standard_error
+  #   @!scope class
+  #   Whether to re-perform a job when a type of +StandardError+ is raised and bubbles up to the GoodJob backend
+  #   @return [Boolean]
   mattr_accessor :reperform_jobs_on_standard_error, default: true
+
+  # @!attribute [rw] on_thread_error
+  #   @!scope class
+  #   Called when a thread raises an error
+  #   @example Send errors to Sentry
+  #     # config/initializers/good_job.rb
+  #
+  #     # With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+  #     GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+  #   @return [#call, nil]
   mattr_accessor :on_thread_error, default: nil
 
+  # Shuts down all execution pools
+  # @param wait [Boolean] whether to wait for shutdown
+  # @return [void]
+  def self.shutdown(wait: true)
+    Notifier.instances.each { |adapter| adapter.shutdown(wait: wait) }
+    Scheduler.instances.each { |scheduler| scheduler.shutdown(wait: wait) }
+  end
+
+  # Tests if execution pools are shut down
+  # @return [Boolean] whether execution pools are shut down
+  def self.shutdown?
+    Notifier.instances.all?(&:shutdown?) && Scheduler.instances.all?(&:shutdown?)
+  end
+
+  # Restarts all execution pools
+  # @return [void]
+  def self.restart
+    Notifier.instances.each(&:restart)
+    Scheduler.instances.each(&:restart)
+  end
+
   ActiveSupport.run_load_hooks(:good_job, self)
 end

data/lib/good_job/adapter.rb

@@ -2,25 +2,27 @@ module GoodJob
   class Adapter
     EXECUTION_MODES = [:async, :external, :inline].freeze
 
-    def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
+    def initialize(execution_mode: nil, queues: nil, max_threads: nil, poll_interval: nil, scheduler: nil, notifier: nil, inline: false)
       if inline && execution_mode.nil?
         ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
         execution_mode = :inline
       end
 
-      configuration = GoodJob::Configuration.new({
-                                                   execution_mode: execution_mode,
-                                                   max_threads: max_threads,
-                                                   poll_interval: poll_interval,
-                                                 },
-                                                 env: ENV)
-
-      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(configuration.execution_mode)
+      configuration = GoodJob::Configuration.new(
+        execution_mode: execution_mode,
+        queues: queues,
+        max_threads: max_threads,
+        poll_interval: poll_interval
+      )
 
       @execution_mode = configuration.execution_mode
+      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(@execution_mode)
 
-      @scheduler = scheduler
-      @scheduler = GoodJob::Scheduler.from_configuration(configuration) if @execution_mode == :async && @scheduler.blank?
+      if @execution_mode == :async # rubocop:disable Style/GuardClause
+        @notifier = notifier || GoodJob::Notifier.new
+        @scheduler = scheduler || GoodJob::Scheduler.from_configuration(configuration)
+        @notifier.recipients << [@scheduler, :create_thread]
+      end
     end
 
     def enqueue(active_job)
@@ -42,12 +44,14 @@ module GoodJob
         end
       end
 
-      @scheduler.create_thread if execute_async?
+      executed_locally = execute_async? && @scheduler.create_thread(queue_name: good_job.queue_name)
+      Notifier.notify(queue_name: good_job.queue_name) unless executed_locally
 
       good_job
     end
 
     def shutdown(wait: true)
+      @notifier&.shutdown(wait: wait)
       @scheduler&.shutdown(wait: wait)
     end
 

data/lib/good_job/cli.rb

@@ -10,16 +10,18 @@ module GoodJob
                   desc: "Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)"
     method_option :queues,
                   type: :string,
-                  banner: "queue1,queue2(;queue3,queue4:5)",
-                  desc: "Queues to work from. Separate multiple queues with commas; separate isolated execution pools with semicolons and threads with colons (default: *)"
+                  banner: "queue1,queue2(;queue3,queue4:5;-queue1,queue2)",
+                  desc: "Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)"
     method_option :poll_interval,
                   type: :numeric,
                   desc: "Interval between polls for available jobs in seconds (default: 1)"
     def start
       set_up_application!
 
-      configuration = Configuration.new(options, env: ENV)
-      scheduler = Scheduler.from_configuration(configuration)
+      notifier = GoodJob::Notifier.new
+      configuration = GoodJob::Configuration.new(options)
+      scheduler = GoodJob::Scheduler.from_configuration(configuration)
+      notifier.recipients << [scheduler, :create_thread]
 
       @stop_good_job_executable = false
       %w[INT TERM].each do |signal|
@@ -28,9 +30,10 @@ module GoodJob
 
       Kernel.loop do
         sleep 0.1
-        break if @stop_good_job_executable || scheduler.shutdown?
+        break if @stop_good_job_executable || scheduler.shutdown? || notifier.shutdown?
       end
 
+      notifier.shutdown
       scheduler.shutdown
     end
 
@@ -41,6 +44,7 @@ module GoodJob
                   type: :numeric,
                   default: 24 * 60 * 60,
                   desc: "Delete records finished more than this many seconds ago"
+
     def cleanup_preserved_jobs
       set_up_application!
 
@@ -55,6 +59,10 @@ module GoodJob
     no_commands do
       def set_up_application!
         require RAILS_ENVIRONMENT_RB
+        return unless defined?(GOOD_JOB_LOG_TO_STDOUT) && GOOD_JOB_LOG_TO_STDOUT && !ActiveSupport::Logger.logger_outputs_to?(GoodJob.logger, STDOUT)
+
+        GoodJob::LogSubscriber.loggers << ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+        GoodJob::LogSubscriber.reset_logger
       end
     end
   end

data/lib/good_job/current_execution.rb

@@ -0,0 +1,37 @@
+require 'active_support/core_ext/module/attribute_accessors_per_thread'
+
+module GoodJob
+  # Thread-local attributes for passing values from Instrumentation.
+  # (Cannot use ActiveSupport::CurrentAttributes because ActiveJob resets it)
+
+  module CurrentExecution
+    # @!attribute [rw] error_on_retry
+    #   @!scope class
+    #   Error captured by retry_on
+    #   @return [Exception, nil]
+    thread_mattr_accessor :error_on_retry
+
+    # @!attribute [rw] error_on_discard
+    #   @!scope class
+    #   Error captured by discard_on
+    #   @return [Exception, nil]
+    thread_mattr_accessor :error_on_discard
+
+    # Resets attributes
+    # @return [void]
+    def self.reset
+      self.error_on_retry = nil
+      self.error_on_discard = nil
+    end
+
+    # @return [Integer] Current process ID
+    def self.process_id
+      Process.pid
+    end
+
+    # @return [String] Current thread name
+    def self.thread_name
+      (Thread.current.name || Thread.current.object_id).to_s
+    end
+  end
+end

data/lib/good_job/job.rb

@@ -9,6 +9,25 @@ module GoodJob
 
     self.table_name = 'good_jobs'.freeze
 
+    def self.queue_parser(string)
+      string = string.presence || '*'
+
+      if string.first == '-'
+        exclude_queues = true
+        string = string[1..-1]
+      end
+
+      queues = string.split(',').map(&:strip)
+
+      if queues.include?('*')
+        { all: true }
+      elsif exclude_queues
+        { exclude: queues }
+      else
+        { include: queues }
+      end
+    end
+
     scope :unfinished, (lambda do
       if column_names.include?('finished_at')
         where(finished_at: nil)
@@ -21,8 +40,15 @@ module GoodJob
     scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
     scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
     scope :queue_string, (lambda do |string|
-      queue_names_without_all = (string.presence || '*').split(',').map(&:strip).reject { |q| q == '*' }
-      where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
+      parsed = queue_parser(string)
+
+      if parsed[:all]
+        all
+      elsif parsed[:exclude]
+        where.not(queue_name: parsed[:exclude]).or where(queue_name: nil)
+      elsif parsed[:include]
+        where(queue_name: parsed[:include])
+      end
     end)
 
     def self.perform_with_advisory_lock
@@ -47,7 +73,7 @@ module GoodJob
           queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
           priority: active_job.priority || DEFAULT_PRIORITY,
           serialized_params: active_job.serialize,
-          scheduled_at: scheduled_at || Time.current,
+          scheduled_at: scheduled_at,
           create_with_advisory_lock: create_with_advisory_lock
         )
 
@@ -63,11 +89,11 @@ module GoodJob
     def perform(destroy_after: !GoodJob.preserve_job_records, reperform_on_standard_error: GoodJob.reperform_jobs_on_standard_error)
       raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
 
+      GoodJob::CurrentExecution.reset
       result = nil
       rescued_error = nil
       error = nil
 
-      ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
       self.performed_at = Time.current
       save! unless destroy_after
 
@@ -76,18 +102,23 @@ module GoodJob
       )
 
       begin
-        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
+        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self, process_id: GoodJob::CurrentExecution.process_id, thread_name: GoodJob::CurrentExecution.thread_name }) do
           result = ActiveJob::Base.execute(params)
         end
       rescue StandardError => e
         rescued_error = e
       end
 
+      retry_or_discard_error = GoodJob::CurrentExecution.error_on_retry ||
+                               GoodJob::CurrentExecution.error_on_discard
+
       if rescued_error
         error = rescued_error
       elsif result.is_a?(Exception)
         error = result
         result = nil
+      elsif retry_or_discard_error
+        error = retry_or_discard_error
       end
 
       error_message = "#{error.class}: #{error.message}" if error

data/lib/good_job/log_subscriber.rb

@@ -32,7 +32,7 @@ module GoodJob
       performer_name = event.payload[:performer_name]
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob started scheduler with queues=#{performer_name} max_threads=#{max_threads} poll_interval=#{poll_interval}."
       end
     end
@@ -40,7 +40,7 @@ module GoodJob
     def scheduler_shutdown_start(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob shutting down scheduler..."
       end
     end
@@ -48,7 +48,7 @@ module GoodJob
     def scheduler_shutdown(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob scheduler is shut down."
       end
     end
@@ -56,24 +56,100 @@ module GoodJob
     def scheduler_restart_pools(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob scheduler has restarted."
       end
     end
 
+    def perform_job(event)
+      good_job = event.payload[:good_job]
+      process_id = event.payload[:process_id]
+      thread_name = event.payload[:thread_name]
+
+      info(tags: [process_id, thread_name]) do
+        "Executed GoodJob #{good_job.id}"
+      end
+    end
+
+    def notifier_listen(_event)
+      info do
+        "Notifier subscribed with LISTEN"
+      end
+    end
+
+    def notifier_notified(event)
+      payload = event.payload[:payload]
+
+      debug do
+        "Notifier received payload: #{payload}"
+      end
+    end
+
+    def notifier_notify_error(event)
+      error = event.payload[:error]
+
+      error do
+        "Notifier errored: #{error}"
+      end
+    end
+
+    def notifier_unlisten(_event)
+      info do
+        "Notifier unsubscribed with UNLISTEN"
+      end
+    end
+
     def cleanup_preserved_jobs(event)
       timestamp = event.payload[:timestamp]
       deleted_records_count = event.payload[:deleted_records_count]
 
-      info_and_stdout do
+      info do
         "GoodJob deleted #{deleted_records_count} preserved #{'job'.pluralize(deleted_records_count)} finished before #{timestamp}."
       end
     end
 
-    private
+    class << self
+      def loggers
+        @_loggers ||= [GoodJob.logger]
+      end
+
+      def logger
+        @_logger ||= begin
+                      logger = Logger.new(StringIO.new)
+                      loggers.each do |each_logger|
+                        logger.extend(ActiveSupport::Logger.broadcast(each_logger))
+                      end
+                      logger
+                    end
+      end
+
+      def reset_logger
+        @_logger = nil
+      end
+    end
 
     def logger
-      GoodJob.logger
+      GoodJob::LogSubscriber.logger
+    end
+
+    private
+
+    def tag_logger(*tags, &block)
+      tags = tags.dup.unshift("GoodJob").compact
+
+      self.class.loggers.inject(block) do |inner, each_logger|
+        if each_logger.respond_to?(:tagged)
+          tags_for_logger = if each_logger.formatter.current_tags.include?("ActiveJob")
+                              ["ActiveJob"] + tags
+                            else
+                              tags
+                            end
+
+          proc { each_logger.tagged(*tags_for_logger, &inner) }
+        else
+          inner
+        end
+      end.call
     end
 
     %w(info debug warn error fatal unknown).each do |level|
@@ -81,30 +157,11 @@ module GoodJob
         def #{level}(progname = nil, tags: [], &block)
           return unless logger
 
-          if logger.respond_to?(:tagged)
-            tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
-            logger.tagged(*tags.compact) do
-              logger.#{level}(progname, &block)
-            end
-          else
+          tag_logger(*tags) do
             logger.#{level}(progname, &block)
           end
         end
       METHOD
     end
-
-    def info_and_stdout(progname = nil, tags: [], &block)
-      unless ActiveSupport::Logger.logger_outputs_to?(logger, STDOUT)
-        tags_string = (['GoodJob'] + tags).map { |t| "[#{t}]" }.join(' ')
-        stdout_message = "#{tags_string} #{yield}"
-        $stdout.puts stdout_message
-      end
-
-      info(progname, tags: [], &block)
-    end
-
-    def thread_name
-      Thread.current.name || Thread.current.object_id
-    end
   end
 end

data/lib/good_job/multi_scheduler.rb

@@ -18,8 +18,17 @@ module GoodJob
       schedulers.each { |s| s.restart(wait: wait) }
     end
 
-    def create_thread
-      schedulers.all?(&:create_thread)
+    def create_thread(state = nil)
+      results = []
+      any_true = schedulers.any? do |scheduler|
+        scheduler.create_thread(state).tap { |result| results << result }
+      end
+
+      if any_true
+        true
+      else
+        results.any? { |result| result == false } ? false : nil
+      end
     end
   end
 end

data/lib/good_job/notifier.rb

@@ -0,0 +1,116 @@
+require 'concurrent/atomic/atomic_boolean'
+
+module GoodJob # :nodoc:
+  #
+  # Wrapper for Postgres LISTEN/NOTIFY
+  #
+  class Notifier
+    CHANNEL = 'good_job'.freeze
+    POOL_OPTIONS = {
+      min_threads: 0,
+      max_threads: 1,
+      auto_terminate: true,
+      idletime: 60,
+      max_queue: 1,
+      fallback_policy: :discard,
+    }.freeze
+    WAIT_INTERVAL = 1
+
+    # @!attribute [r] instances
+    #   @!scope class
+    #   @return [array<GoodJob:Adapter>] the instances of +GoodJob::Notifier+
+    cattr_reader :instances, default: [], instance_reader: false
+
+    def self.notify(message)
+      connection = ActiveRecord::Base.connection
+      connection.exec_query <<~SQL
+        NOTIFY #{CHANNEL}, #{connection.quote(message.to_json)}
+      SQL
+    end
+
+    attr_reader :recipients
+
+    def initialize(*recipients)
+      @recipients = Concurrent::Array.new(recipients)
+      @listening = Concurrent::AtomicBoolean.new(false)
+
+      self.class.instances << self
+
+      create_pool
+      listen
+    end
+
+    def listening?
+      @listening.true?
+    end
+
+    def restart(wait: true)
+      shutdown(wait: wait)
+      create_pool
+      listen
+    end
+
+    def shutdown(wait: true)
+      return unless @pool.running?
+
+      @pool.shutdown
+      @pool.wait_for_termination if wait
+    end
+
+    def shutdown?
+      !@pool.running?
+    end
+
+    private
+
+    def create_pool
+      @pool = Concurrent::ThreadPoolExecutor.new(POOL_OPTIONS)
+    end
+
+    def listen
+      future = Concurrent::Future.new(args: [@recipients, @pool, @listening], executor: @pool) do |recipients, pool, listening|
+        Rails.application.reloader.wrap do
+          conn = ActiveRecord::Base.connection.raw_connection
+          ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
+            conn.async_exec "LISTEN #{CHANNEL}"
+          end
+
+          begin
+            ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
+              while pool.running?
+                listening.make_true
+                conn.wait_for_notify(WAIT_INTERVAL) do |channel, _pid, payload|
+                  listening.make_false
+                  next unless channel == CHANNEL
+
+                  ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
+                  parsed_payload = JSON.parse(payload, symbolize_names: true)
+                  recipients.each do |recipient|
+                    target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
+                    target.send(method_name, parsed_payload)
+                  end
+                end
+                listening.make_false
+              end
+            end
+          rescue StandardError => e
+            ActiveSupport::Notifications.instrument("notifier_notify_error.good_job", { error: e })
+            raise
+          ensure
+            @listening.make_false
+            ActiveSupport::Notifications.instrument("notifier_unlisten.good_job") do
+              conn.async_exec "UNLISTEN *"
+            end
+          end
+        end
+      end
+
+      future.add_observer(self, :listen_observer)
+      future.execute
+    end
+
+    def listen_observer(_time, _result, _thread_error)
+      listen unless shutdown?
+    end
+  end
+end

data/lib/good_job/performer.rb

@@ -2,14 +2,21 @@ module GoodJob
   class Performer
     attr_reader :name
 
-    def initialize(target, method_name, name: nil)
+    def initialize(target, method_name, name: nil, filter: nil)
       @target = target
       @method_name = method_name
       @name = name
+      @filter = filter
     end
 
     def next
       @target.public_send(@method_name)
     end
+
+    def next?(state = {})
+      return true unless @filter.respond_to?(:call)
+
+      @filter.call(state)
+    end
   end
 end

data/lib/good_job/railtie.rb

@@ -4,5 +4,15 @@ module GoodJob
       ActiveSupport.on_load(:good_job) { self.logger = ::Rails.logger }
       GoodJob::LogSubscriber.attach_to :good_job
     end
+
+    initializer "good_job.active_job_notifications" do
+      ActiveSupport::Notifications.subscribe "enqueue_retry.active_job" do |event|
+        GoodJob::CurrentExecution.error_on_retry = event.payload[:error]
+      end
+
+      ActiveSupport::Notifications.subscribe "discard.active_job" do |event|
+        GoodJob::CurrentExecution.error_on_discard = event.payload[:error]
+      end
+    end
   end
 end

data/lib/good_job/scheduler.rb

@@ -2,16 +2,25 @@ require "concurrent/executor/thread_pool_executor"
 require "concurrent/timer_task"
 require "concurrent/utility/processor_counter"
 
-module GoodJob
+module GoodJob # :nodoc:
+  #
+  # Schedulers are generic thread execution pools that are responsible for
+  # periodically checking for available execution tasks, executing tasks in a
+  # bounded thread-pool, and efficiently scaling execution threads.
+  #
+  # Schedulers are "generic" in the sense that they delegate task execution
+  # details to a "Performer" object that responds to #next.
+  #
   class Scheduler
+    # Defaults for instance of Concurrent::TimerTask
     DEFAULT_TIMER_OPTIONS = {
       execution_interval: 1,
       timeout_interval: 1,
       run_now: true,
     }.freeze
 
+    # Defaults for instance of Concurrent::ThreadPoolExecutor
     DEFAULT_POOL_OPTIONS = {
-      name: 'good_job',
       min_threads: 0,
       max_threads: Concurrent.processor_count,
       auto_terminate: true,
@@ -20,18 +29,35 @@ module GoodJob
       fallback_policy: :discard,
     }.freeze
 
+    # @!attribute [r] instances
+    #   @!scope class
+    #   All instantiated Schedulers in the current process.
+    #   @return [array<GoodJob:Scheduler>]
     cattr_reader :instances, default: [], instance_reader: false
 
+    # Creates GoodJob::Scheduler(s) and Performers from a GoodJob::Configuration instance.
+    # @param configuration [GoodJob::Configuration]
+    # @return [GoodJob::Scheduler, GoodJob::MultiScheduler]
     def self.from_configuration(configuration)
       schedulers = configuration.queue_string.split(';').map do |queue_string_and_max_threads|
         queue_string, max_threads = queue_string_and_max_threads.split(':')
         max_threads = (max_threads || configuration.max_threads).to_i
 
         job_query = GoodJob::Job.queue_string(queue_string)
-        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
+        parsed = GoodJob::Job.queue_parser(queue_string)
+        job_filter = proc do |state|
+          if parsed[:exclude]
+            !parsed[:exclude].include? state[:queue_name]
+          elsif parsed[:include]
+            parsed[:include].include? state[:queue_name]
+          else
+            true
+          end
+        end
+        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string, filter: job_filter)
 
         timer_options = {}
-        timer_options[:execution_interval] = configuration.poll_interval if configuration.poll_interval.positive?
+        timer_options[:execution_interval] = configuration.poll_interval
 
         pool_options = {
           max_threads: max_threads,
@@ -47,6 +73,9 @@ module GoodJob
       end
     end
 
+    # @param performer [GoodJob::Performer]
+    # @param timer_options [Hash] Options to instantiate a Concurrent::TimerTask
+    # @param pool_options [Hash] Options to instantiate a Concurrent::ThreadPoolExecutor
     def initialize(performer, timer_options: {}, pool_options: {})
       raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 
@@ -56,14 +85,19 @@ module GoodJob
       @pool_options = DEFAULT_POOL_OPTIONS.merge(pool_options)
       @timer_options = DEFAULT_TIMER_OPTIONS.merge(timer_options)
 
+      @pool_options[:name] = "GoodJob::Scheduler(queues=#{@performer.name} max_threads=#{@pool_options[:max_threads]} poll_interval=#{@timer_options[:execution_interval]})"
+
       create_pools
     end
 
+    # Shut down the Scheduler.
+    # @param wait [Boolean] Wait for actively executing jobs to finish
+    # @return [void]
     def shutdown(wait: true)
       @_shutdown = true
 
-      ActiveSupport::Notifications.instrument("scheduler_shutdown_start.good_job", { wait: wait, process_id: process_id })
-      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait, process_id: process_id }) do
+      instrument("scheduler_shutdown_start", { wait: wait })
+      instrument("scheduler_shutdown", { wait: wait }) do
         if @timer&.running?
           @timer.shutdown
           @timer.wait_for_termination if wait
@@ -76,19 +110,32 @@ module GoodJob
       end
     end
 
+    # True when the Scheduler is shutdown.
+    # @return [true, false, nil]
     def shutdown?
       @_shutdown
     end
 
+    # Restart the Scheduler. When shutdown, start; or shutdown and start.
+    # @param wait [Boolean] Wait for actively executing jobs to finish
+    # @return [void]
     def restart(wait: true)
-      ActiveSupport::Notifications.instrument("scheduler_restart_pools.good_job", { process_id: process_id }) do
+      instrument("scheduler_restart_pools") do
         shutdown(wait: wait) unless shutdown?
         create_pools
+        @_shutdown = false
       end
     end
 
-    def create_thread
-      return false unless @pool.ready_worker_count.positive?
+    # Triggers a Performer execution, if an execution thread is available.
+    # @param state [nil, Object] Allows Performer#next? to accept or reject the execution
+    # @return [nil, Boolean] if the thread was created
+    def create_thread(state = nil)
+      return nil unless @pool.running? && @pool.ready_worker_count.positive?
+
+      if state
+        return false unless @performer.next?(state)
+      end
 
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
         output = nil
@@ -97,35 +144,32 @@ module GoodJob
       end
       future.add_observer(self, :task_observer)
       future.execute
+
+      true
     end
 
+    # Invoked on completion of TimerTask task.
+    # @!visibility private
+    # @return [void]
     def timer_observer(time, executed_task, thread_error)
       GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
-      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
+      instrument("finished_timer_task", { result: executed_task, error: thread_error, time: time })
     end
 
+    # Invoked on completion of ThreadPoolExecutor task
+    # @!visibility private
+    # @return [void]
     def task_observer(time, output, thread_error)
       GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
-      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
+      instrument("finished_job_task", { result: output, error: thread_error, time: time })
       create_thread if output
     end
 
-    class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
-      # https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
-      def ready_worker_count
-        synchronize do
-          workers_still_to_be_created = @max_length - @pool.length
-          workers_created_but_waiting = @ready.length
-
-          workers_still_to_be_created + workers_created_but_waiting
-        end
-      end
-    end
-
     private
 
+    # @return [void]
     def create_pools
-      ActiveSupport::Notifications.instrument("scheduler_create_pools.good_job", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval], process_id: process_id }) do
+      instrument("scheduler_create_pools", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval] }) do
         @pool = ThreadPoolExecutor.new(@pool_options)
         next unless @timer_options[:execution_interval].positive?
 
@@ -135,12 +179,29 @@ module GoodJob
       end
     end
 
-    def process_id
-      Process.pid
+    def instrument(name, payload = {}, &block)
+      payload = payload.reverse_merge({
+                                        scheduler: self,
+                                        process_id: GoodJob::CurrentExecution.process_id,
+                                        thread_name: GoodJob::CurrentExecution.thread_name,
+                                      })
+
+      ActiveSupport::Notifications.instrument("#{name}.good_job", payload, &block)
     end
+  end
 
-    def thread_name
-      Thread.current.name || Thread.current.object_id
+  # Slightly customized sub-class of Concurrent::ThreadPoolExecutor
+  class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
+    # Number of idle or potential threads available to execute tasks
+    # https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
+    # @return [Integer]
+    def ready_worker_count
+      synchronize do
+        workers_still_to_be_created = @max_length - @pool.length
+        workers_created_but_waiting = @ready.length
+
+        workers_still_to_be_created + workers_created_but_waiting
+      end
     end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '1.1.1'.freeze
+  VERSION = '1.2.1'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.2.1
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-08-12 00:00:00.000000000 Z
+date: 2020-08-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -151,7 +151,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: pry
+  name: pry-rails
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -178,6 +178,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rbtrace
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
@@ -248,6 +262,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: sigdump
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
 email:
 - bensheldon@gmail.com
@@ -270,10 +312,12 @@ files:
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
 - lib/good_job/configuration.rb
+- lib/good_job/current_execution.rb
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
 - lib/good_job/log_subscriber.rb
 - lib/good_job/multi_scheduler.rb
+- lib/good_job/notifier.rb
 - lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb