good_job 1.1.3 → 1.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 44da5dd6eb7dee7e08319f7405e97f81e4223f8c88d48ba4a0512fa006f89e78
-  data.tar.gz: fe3f5ca8a39b85b1aa4be77a819e910691223cf36e80c6ab337ff1b224a26e16
+  metadata.gz: 7134108a565e8b1efaa825cec90630cab0965fcfca5ff02f09751730f8839bdc
+  data.tar.gz: d887e3b6cbb6d3d877793b186de2c0ca5a65178b968b2000821c6f9000c0fdd1
 SHA512:
-  metadata.gz: e6648e41c7ff99915716702651cd43fb7fffcff24528a9fb2d7e4a7913303d85d25c3b3f8ef54cc1b716a273836bc3c7fb2911750e1d99f216b2074b03f13ae8
-  data.tar.gz: '063048f09b76b10d4f38beb5fcf390d972f0952ed4df7b1053e8078da2bad005b816e21271026b95f8eaef12b8e3e1ce0a4cecf2cf40f965e852a8e2e9854aac'
+  metadata.gz: cce4a02b62ded08da0b2665cb346bfbeda6fdffa15182c8ab680962f6ba1f3534a8b9ac4cad8ecfa60d34b15cdfaabff5c8c80e4b3f7bdaa99c2d7a0e4208fe9
+  data.tar.gz: d9f13feb39a8700fe889c7b4250004095cdddeb5e1e9f06426610d10c13ce7c14d20db32810aa93dd4d1b7a075f585821bf4581634373a3585540bab0018f46c

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [v1.1.4](https://github.com/bensheldon/good_job/tree/v1.1.4) (2020-08-19)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.3...v1.1.4)
+
+**Implemented enhancements:**
+
+- Explicitly name threads for easier debugging [\#64](https://github.com/bensheldon/good_job/issues/64)
+- Investigate Listen/Notify as alternative to polling [\#54](https://github.com/bensheldon/good_job/issues/54)
+
+**Merged pull requests:**
+
+- Add Postgres LISTEN/NOTIFY support [\#82](https://github.com/bensheldon/good_job/pull/82) ([bensheldon](https://github.com/bensheldon))
+- Allow Schedulers to filter \#create\_thread to avoid flood of queries when running async with multiple schedulers [\#81](https://github.com/bensheldon/good_job/pull/81) ([bensheldon](https://github.com/bensheldon))
+- Fully name scheduler threadpools and thread names; refactor CLI STDOUT [\#80](https://github.com/bensheldon/good_job/pull/80) ([bensheldon](https://github.com/bensheldon))
+
 ## [v1.1.3](https://github.com/bensheldon/good_job/tree/v1.1.3) (2020-08-14)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.2...v1.1.3)
@@ -36,7 +51,6 @@
 **Merged pull requests:**
 
 - Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
-- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
 
 ## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-10)
 
@@ -71,9 +85,9 @@
 
 **Merged pull requests:**
 
+- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
 - Re-perform a job if a StandardError bubbles up; better document job reliability [\#62](https://github.com/bensheldon/good_job/pull/62) ([bensheldon](https://github.com/bensheldon))
 - Update the setup documentation to use correct bin setup command [\#61](https://github.com/bensheldon/good_job/pull/61) ([jm96441n](https://github.com/jm96441n))
-- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
 
 ## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-25)
 
@@ -94,7 +108,7 @@
 
 **Merged pull requests:**
 
-- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
 
 ## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
 
@@ -131,6 +145,7 @@
 
 **Merged pull requests:**
 
+- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
 - Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-16)

data/README.md

@@ -6,7 +6,7 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 
 - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration. 
 - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby). 
-- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
+- **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
 For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
@@ -284,15 +284,20 @@ Depending on your application configuration, you may need to take additional ste
     # config/puma.rb
   
     before_fork do
-      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+      GoodJob.shutdown
     end
     
     on_worker_boot do
-      GoodJob::Scheduler.instances.each { |s| s.restart }
+      GoodJob.restart
     end
     
     on_worker_shutdown do
-      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+      GoodJob.shutdown
+    end
+  
+    MAIN_PID = Process.pid
+    at_exit do
+      GoodJob.shutdown if Process.pid == MAIN_PID
     end
     ```
   

data/exe/good_job

@@ -1,3 +1,4 @@
 #!/usr/bin/env ruby
 require 'good_job/cli'
+GOOD_JOB_LOG_TO_STDOUT = true
 GoodJob::CLI.start(ARGV)

data/lib/good_job.rb

@@ -11,14 +11,36 @@ require 'good_job/adapter'
 require 'good_job/pg_locks'
 require 'good_job/performer'
 require 'good_job/current_execution'
+require 'good_job/notifier'
 
 require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
-  cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+  mattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
   mattr_accessor :preserve_job_records, default: false
   mattr_accessor :reperform_jobs_on_standard_error, default: true
   mattr_accessor :on_thread_error, default: nil
 
   ActiveSupport.run_load_hooks(:good_job, self)
+
+  # Shuts down all execution pools
+  # @param wait [Boolean] whether to wait for shutdown
+  # @return [void]
+  def self.shutdown(wait: true)
+    Notifier.instances.each { |adapter| adapter.shutdown(wait: wait) }
+    Scheduler.instances.each { |scheduler| scheduler.shutdown(wait: wait) }
+  end
+
+  # Tests if execution pools are shut down
+  # @return [Boolean] whether execution pools are shut down
+  def self.shutdown?
+    Notifier.instances.all?(&:shutdown?) && Scheduler.instances.all?(&:shutdown?)
+  end
+
+  # Restarts all execution pools
+  # @return [void]
+  def self.restart
+    Notifier.instances.each(&:restart)
+    Scheduler.instances.each(&:restart)
+  end
 end

data/lib/good_job/adapter.rb

@@ -2,25 +2,27 @@ module GoodJob
   class Adapter
     EXECUTION_MODES = [:async, :external, :inline].freeze
 
-    def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
+    def initialize(execution_mode: nil, queues: nil, max_threads: nil, poll_interval: nil, scheduler: nil, notifier: nil, inline: false)
       if inline && execution_mode.nil?
         ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
         execution_mode = :inline
       end
 
-      configuration = GoodJob::Configuration.new({
-                                                   execution_mode: execution_mode,
-                                                   max_threads: max_threads,
-                                                   poll_interval: poll_interval,
-                                                 },
-                                                 env: ENV)
-
-      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(configuration.execution_mode)
+      configuration = GoodJob::Configuration.new(
+        execution_mode: execution_mode,
+        queues: queues,
+        max_threads: max_threads,
+        poll_interval: poll_interval
+      )
 
       @execution_mode = configuration.execution_mode
+      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(@execution_mode)
 
-      @scheduler = scheduler
-      @scheduler = GoodJob::Scheduler.from_configuration(configuration) if @execution_mode == :async && @scheduler.blank?
+      if @execution_mode == :async # rubocop:disable Style/GuardClause
+        @notifier = notifier || GoodJob::Notifier.new
+        @scheduler = scheduler || GoodJob::Scheduler.from_configuration(configuration)
+        @notifier.recipients << [@scheduler, :create_thread]
+      end
     end
 
     def enqueue(active_job)
@@ -42,12 +44,14 @@ module GoodJob
         end
       end
 
-      @scheduler.create_thread if execute_async?
+      executed_locally = execute_async? && @scheduler.create_thread(queue_name: good_job.queue_name)
+      Notifier.notify(queue_name: good_job.queue_name) unless executed_locally
 
       good_job
     end
 
     def shutdown(wait: true)
+      @notifier&.shutdown(wait: wait)
       @scheduler&.shutdown(wait: wait)
     end
 

data/lib/good_job/cli.rb

@@ -18,8 +18,10 @@ module GoodJob
     def start
       set_up_application!
 
-      configuration = Configuration.new(options, env: ENV)
-      scheduler = Scheduler.from_configuration(configuration)
+      notifier = GoodJob::Notifier.new
+      configuration = GoodJob::Configuration.new(options)
+      scheduler = GoodJob::Scheduler.from_configuration(configuration)
+      notifier.recipients << [scheduler, :create_thread]
 
       @stop_good_job_executable = false
       %w[INT TERM].each do |signal|
@@ -28,9 +30,10 @@ module GoodJob
 
       Kernel.loop do
         sleep 0.1
-        break if @stop_good_job_executable || scheduler.shutdown?
+        break if @stop_good_job_executable || scheduler.shutdown? || notifier.shutdown?
       end
 
+      notifier.shutdown
       scheduler.shutdown
     end
 
@@ -41,6 +44,7 @@ module GoodJob
                   type: :numeric,
                   default: 24 * 60 * 60,
                   desc: "Delete records finished more than this many seconds ago"
+
     def cleanup_preserved_jobs
       set_up_application!
 
@@ -55,6 +59,10 @@ module GoodJob
     no_commands do
       def set_up_application!
         require RAILS_ENVIRONMENT_RB
+        return unless defined?(GOOD_JOB_LOG_TO_STDOUT) && GOOD_JOB_LOG_TO_STDOUT && !ActiveSupport::Logger.logger_outputs_to?(GoodJob.logger, STDOUT)
+
+        GoodJob::LogSubscriber.loggers << ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
+        GoodJob::LogSubscriber.reset_logger
       end
     end
   end

data/lib/good_job/current_execution.rb

@@ -21,5 +21,15 @@ module GoodJob
       self.error_on_retry = nil
       self.error_on_discard = nil
     end
+
+    # @return [Integer] Current process ID
+    def self.process_id
+      Process.pid
+    end
+
+    # @return [String] Current thread name
+    def self.thread_name
+      (Thread.current.name || Thread.current.object_id).to_s
+    end
   end
 end

data/lib/good_job/job.rb

@@ -9,6 +9,25 @@ module GoodJob
 
     self.table_name = 'good_jobs'.freeze
 
+    def self.queue_parser(string)
+      string = string.presence || '*'
+
+      if string.first == '-'
+        exclude_queues = true
+        string = string[1..-1]
+      end
+
+      queues = string.split(',').map(&:strip)
+
+      if queues.include?('*')
+        { all: true }
+      elsif exclude_queues
+        { exclude: queues }
+      else
+        { include: queues }
+      end
+    end
+
     scope :unfinished, (lambda do
       if column_names.include?('finished_at')
         where(finished_at: nil)
@@ -21,20 +40,14 @@ module GoodJob
     scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
     scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
     scope :queue_string, (lambda do |string|
-      string = string.presence || '*'
-
-      if string.first == '-'
-        exclude_queues = true
-        string = string[1..-1]
-      end
-
-      queue_names_without_all = string.split(',').map(&:strip).reject { |q| q == '*' }
-      return if queue_names_without_all.size.zero?
-
-      if exclude_queues
-        where.not(queue_name: queue_names_without_all).or where(queue_name: nil)
-      else
-        where(queue_name: queue_names_without_all)
+      parsed = queue_parser(string)
+
+      if parsed[:all]
+        all
+      elsif parsed[:exclude]
+        where.not(queue_name: parsed[:exclude]).or where(queue_name: nil)
+      elsif parsed[:include]
+        where(queue_name: parsed[:include])
       end
     end)
 
@@ -60,7 +73,7 @@ module GoodJob
           queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
           priority: active_job.priority || DEFAULT_PRIORITY,
           serialized_params: active_job.serialize,
-          scheduled_at: scheduled_at || Time.current,
+          scheduled_at: scheduled_at,
           create_with_advisory_lock: create_with_advisory_lock
         )
 
@@ -89,7 +102,7 @@ module GoodJob
       )
 
       begin
-        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
+        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self, process_id: GoodJob::CurrentExecution.process_id, thread_name: GoodJob::CurrentExecution.thread_name }) do
           result = ActiveJob::Base.execute(params)
         end
       rescue StandardError => e

data/lib/good_job/log_subscriber.rb

@@ -32,7 +32,7 @@ module GoodJob
       performer_name = event.payload[:performer_name]
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob started scheduler with queues=#{performer_name} max_threads=#{max_threads} poll_interval=#{poll_interval}."
       end
     end
@@ -40,7 +40,7 @@ module GoodJob
     def scheduler_shutdown_start(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob shutting down scheduler..."
       end
     end
@@ -48,7 +48,7 @@ module GoodJob
     def scheduler_shutdown(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob scheduler is shut down."
       end
     end
@@ -56,24 +56,100 @@ module GoodJob
     def scheduler_restart_pools(event)
       process_id = event.payload[:process_id]
 
-      info_and_stdout(tags: [process_id]) do
+      info(tags: [process_id]) do
         "GoodJob scheduler has restarted."
       end
     end
 
+    def perform_job(event)
+      good_job = event.payload[:good_job]
+      process_id = event.payload[:process_id]
+      thread_name = event.payload[:thread_name]
+
+      info(tags: [process_id, thread_name]) do
+        "Executed GoodJob #{good_job.id}"
+      end
+    end
+
+    def notifier_listen(_event)
+      info do
+        "Notifier subscribed with LISTEN"
+      end
+    end
+
+    def notifier_notified(event)
+      payload = event.payload[:payload]
+
+      debug do
+        "Notifier received payload: #{payload}"
+      end
+    end
+
+    def notifier_notify_error(event)
+      error = event.payload[:error]
+
+      error do
+        "Notifier errored: #{error}"
+      end
+    end
+
+    def notifier_unlisten(_event)
+      info do
+        "Notifier unsubscribed with UNLISTEN"
+      end
+    end
+
     def cleanup_preserved_jobs(event)
       timestamp = event.payload[:timestamp]
       deleted_records_count = event.payload[:deleted_records_count]
 
-      info_and_stdout do
+      info do
         "GoodJob deleted #{deleted_records_count} preserved #{'job'.pluralize(deleted_records_count)} finished before #{timestamp}."
       end
     end
 
-    private
+    class << self
+      def loggers
+        @_loggers ||= [GoodJob.logger]
+      end
+
+      def logger
+        @_logger ||= begin
+                      logger = Logger.new(StringIO.new)
+                      loggers.each do |each_logger|
+                        logger.extend(ActiveSupport::Logger.broadcast(each_logger))
+                      end
+                      logger
+                    end
+      end
+
+      def reset_logger
+        @_logger = nil
+      end
+    end
 
     def logger
-      GoodJob.logger
+      GoodJob::LogSubscriber.logger
+    end
+
+    private
+
+    def tag_logger(*tags, &block)
+      tags = tags.dup.unshift("GoodJob").compact
+
+      self.class.loggers.inject(block) do |inner, each_logger|
+        if each_logger.respond_to?(:tagged)
+          tags_for_logger = if each_logger.formatter.current_tags.include?("ActiveJob")
+                              ["ActiveJob"] + tags
+                            else
+                              tags
+                            end
+
+          proc { each_logger.tagged(*tags_for_logger, &inner) }
+        else
+          inner
+        end
+      end.call
     end
 
     %w(info debug warn error fatal unknown).each do |level|
@@ -81,30 +157,11 @@ module GoodJob
         def #{level}(progname = nil, tags: [], &block)
           return unless logger
 
-          if logger.respond_to?(:tagged)
-            tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
-            logger.tagged(*tags.compact) do
-              logger.#{level}(progname, &block)
-            end
-          else
+          tag_logger(*tags) do
             logger.#{level}(progname, &block)
           end
         end
       METHOD
     end
-
-    def info_and_stdout(progname = nil, tags: [], &block)
-      unless ActiveSupport::Logger.logger_outputs_to?(logger, STDOUT)
-        tags_string = (['GoodJob'] + tags).map { |t| "[#{t}]" }.join(' ')
-        stdout_message = "#{tags_string} #{yield}"
-        $stdout.puts stdout_message
-      end
-
-      info(progname, tags: [], &block)
-    end
-
-    def thread_name
-      Thread.current.name || Thread.current.object_id
-    end
   end
 end

data/lib/good_job/multi_scheduler.rb

@@ -18,8 +18,17 @@ module GoodJob
       schedulers.each { |s| s.restart(wait: wait) }
     end
 
-    def create_thread
-      schedulers.all?(&:create_thread)
+    def create_thread(state = nil)
+      results = []
+      any_true = schedulers.any? do |scheduler|
+        scheduler.create_thread(state).tap { |result| results << result }
+      end
+
+      if any_true
+        true
+      else
+        results.any? { |result| result == false } ? false : nil
+      end
     end
   end
 end

data/lib/good_job/notifier.rb

@@ -0,0 +1,116 @@
+require 'concurrent/atomic/atomic_boolean'
+
+module GoodJob # :nodoc:
+  #
+  # Wrapper for Postgres LISTEN/NOTIFY
+  #
+  class Notifier
+    CHANNEL = 'good_job'.freeze
+    POOL_OPTIONS = {
+      min_threads: 0,
+      max_threads: 1,
+      auto_terminate: true,
+      idletime: 60,
+      max_queue: 1,
+      fallback_policy: :discard,
+    }.freeze
+    WAIT_INTERVAL = 1
+
+    # @!attribute [r] instances
+    #   @!scope class
+    #   @return [array<GoodJob:Adapter>] the instances of +GoodJob::Notifier+
+    cattr_reader :instances, default: [], instance_reader: false
+
+    def self.notify(message)
+      connection = ActiveRecord::Base.connection
+      connection.exec_query <<~SQL
+        NOTIFY #{CHANNEL}, #{connection.quote(message.to_json)}
+      SQL
+    end
+
+    attr_reader :recipients
+
+    def initialize(*recipients)
+      @recipients = Concurrent::Array.new(recipients)
+      @listening = Concurrent::AtomicBoolean.new(false)
+
+      self.class.instances << self
+
+      create_pool
+      listen
+    end
+
+    def listening?
+      @listening.true?
+    end
+
+    def restart(wait: true)
+      shutdown(wait: wait)
+      create_pool
+      listen
+    end
+
+    def shutdown(wait: true)
+      return unless @pool.running?
+
+      @pool.shutdown
+      @pool.wait_for_termination if wait
+    end
+
+    def shutdown?
+      !@pool.running?
+    end
+
+    private
+
+    def create_pool
+      @pool = Concurrent::ThreadPoolExecutor.new(POOL_OPTIONS)
+    end
+
+    def listen
+      future = Concurrent::Future.new(args: [@recipients, @pool, @listening], executor: @pool) do |recipients, pool, listening|
+        Rails.application.reloader.wrap do
+          conn = ActiveRecord::Base.connection.raw_connection
+          ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
+            conn.async_exec "LISTEN #{CHANNEL}"
+          end
+
+          begin
+            ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
+              while pool.running?
+                listening.make_true
+                conn.wait_for_notify(WAIT_INTERVAL) do |channel, _pid, payload|
+                  listening.make_false
+                  next unless channel == CHANNEL
+
+                  ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
+                  parsed_payload = JSON.parse(payload, symbolize_names: true)
+                  recipients.each do |recipient|
+                    target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
+                    target.send(method_name, parsed_payload)
+                  end
+                end
+                listening.make_false
+              end
+            end
+          rescue StandardError => e
+            ActiveSupport::Notifications.instrument("notifier_notify_error.good_job", { error: e })
+            raise
+          ensure
+            @listening.make_false
+            ActiveSupport::Notifications.instrument("notifier_unlisten.good_job") do
+              conn.async_exec "UNLISTEN *"
+            end
+          end
+        end
+      end
+
+      future.add_observer(self, :listen_observer)
+      future.execute
+    end
+
+    def listen_observer(_time, _result, _thread_error)
+      listen unless shutdown?
+    end
+  end
+end

data/lib/good_job/performer.rb

@@ -2,14 +2,21 @@ module GoodJob
   class Performer
     attr_reader :name
 
-    def initialize(target, method_name, name: nil)
+    def initialize(target, method_name, name: nil, filter: nil)
       @target = target
       @method_name = method_name
       @name = name
+      @filter = filter
     end
 
     def next
       @target.public_send(@method_name)
     end
+
+    def next?(state = {})
+      return true unless @filter.respond_to?(:call)
+
+      @filter.call(state)
+    end
   end
 end

data/lib/good_job/scheduler.rb

@@ -3,12 +3,14 @@ require "concurrent/timer_task"
 require "concurrent/utility/processor_counter"
 
 module GoodJob # :nodoc:
+  #
   # Schedulers are generic thread execution pools that are responsible for
   # periodically checking for available execution tasks, executing tasks in a
   # bounded thread-pool, and efficiently scaling execution threads.
   #
   # Schedulers are "generic" in the sense that they delegate task execution
   # details to a "Performer" object that responds to #next.
+  #
   class Scheduler
     # Defaults for instance of Concurrent::TimerTask
     DEFAULT_TIMER_OPTIONS = {
@@ -19,7 +21,6 @@ module GoodJob # :nodoc:
 
     # Defaults for instance of Concurrent::ThreadPoolExecutor
     DEFAULT_POOL_OPTIONS = {
-      name: 'good_job',
       min_threads: 0,
       max_threads: Concurrent.processor_count,
       auto_terminate: true,
@@ -43,10 +44,20 @@ module GoodJob # :nodoc:
         max_threads = (max_threads || configuration.max_threads).to_i
 
         job_query = GoodJob::Job.queue_string(queue_string)
-        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
+        parsed = GoodJob::Job.queue_parser(queue_string)
+        job_filter = proc do |state|
+          if parsed[:exclude]
+            !parsed[:exclude].include? state[:queue_name]
+          elsif parsed[:include]
+            parsed[:include].include? state[:queue_name]
+          else
+            true
+          end
+        end
+        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string, filter: job_filter)
 
         timer_options = {}
-        timer_options[:execution_interval] = configuration.poll_interval if configuration.poll_interval.positive?
+        timer_options[:execution_interval] = configuration.poll_interval
 
         pool_options = {
           max_threads: max_threads,
@@ -74,6 +85,8 @@ module GoodJob # :nodoc:
       @pool_options = DEFAULT_POOL_OPTIONS.merge(pool_options)
       @timer_options = DEFAULT_TIMER_OPTIONS.merge(timer_options)
 
+      @pool_options[:name] = "GoodJob::Scheduler(queues=#{@performer.name} max_threads=#{@pool_options[:max_threads]} poll_interval=#{@timer_options[:execution_interval]})"
+
       create_pools
     end
 
@@ -83,8 +96,8 @@ module GoodJob # :nodoc:
     def shutdown(wait: true)
       @_shutdown = true
 
-      ActiveSupport::Notifications.instrument("scheduler_shutdown_start.good_job", { wait: wait, process_id: process_id })
-      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait, process_id: process_id }) do
+      instrument("scheduler_shutdown_start", { wait: wait })
+      instrument("scheduler_shutdown", { wait: wait }) do
         if @timer&.running?
           @timer.shutdown
           @timer.wait_for_termination if wait
@@ -107,16 +120,22 @@ module GoodJob # :nodoc:
     # @param wait [Boolean] Wait for actively executing jobs to finish
     # @return [void]
     def restart(wait: true)
-      ActiveSupport::Notifications.instrument("scheduler_restart_pools.good_job", { process_id: process_id }) do
+      instrument("scheduler_restart_pools") do
         shutdown(wait: wait) unless shutdown?
         create_pools
+        @_shutdown = false
       end
     end
 
-    # Triggers the execution the Performer, if an execution thread is available.
-    # @return [Boolean]
-    def create_thread
-      return false unless @pool.ready_worker_count.positive?
+    # Triggers a Performer execution, if an execution thread is available.
+    # @param state [nil, Object] Allows Performer#next? to accept or reject the execution
+    # @return [nil, Boolean] if the thread was created
+    def create_thread(state = nil)
+      return nil unless @pool.running? && @pool.ready_worker_count.positive?
+
+      if state
+        return false unless @performer.next?(state)
+      end
 
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
         output = nil
@@ -125,6 +144,7 @@ module GoodJob # :nodoc:
       end
       future.add_observer(self, :task_observer)
       future.execute
+
       true
     end
 
@@ -133,7 +153,7 @@ module GoodJob # :nodoc:
     # @return [void]
     def timer_observer(time, executed_task, thread_error)
       GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
-      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
+      instrument("finished_timer_task", { result: executed_task, error: thread_error, time: time })
     end
 
     # Invoked on completion of ThreadPoolExecutor task
@@ -141,7 +161,7 @@ module GoodJob # :nodoc:
     # @return [void]
     def task_observer(time, output, thread_error)
       GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
-      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
+      instrument("finished_job_task", { result: output, error: thread_error, time: time })
       create_thread if output
     end
 
@@ -149,7 +169,7 @@ module GoodJob # :nodoc:
 
     # @return [void]
     def create_pools
-      ActiveSupport::Notifications.instrument("scheduler_create_pools.good_job", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval], process_id: process_id }) do
+      instrument("scheduler_create_pools", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval] }) do
         @pool = ThreadPoolExecutor.new(@pool_options)
         next unless @timer_options[:execution_interval].positive?
 
@@ -159,14 +179,14 @@ module GoodJob # :nodoc:
       end
     end
 
-    # @return [Integer] Current process ID
-    def process_id
-      Process.pid
-    end
+    def instrument(name, payload = {}, &block)
+      payload = payload.reverse_merge({
+                                        scheduler: self,
+                                        process_id: GoodJob::CurrentExecution.process_id,
+                                        thread_name: GoodJob::CurrentExecution.thread_name,
+                                      })
 
-    # @return [String] Current thread name
-    def thread_name
-      (Thread.current.name || Thread.current.object_id).to_s
+      ActiveSupport::Notifications.instrument("#{name}.good_job", payload, &block)
     end
   end
 

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '1.1.3'.freeze
+  VERSION = '1.1.4'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.1.3
+  version: 1.1.4
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-08-14 00:00:00.000000000 Z
+date: 2020-08-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -151,7 +151,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: pry
+  name: pry-rails
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -178,6 +178,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rbtrace
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
@@ -248,6 +262,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: sigdump
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: yard
   requirement: !ruby/object:Gem::Requirement
@@ -289,6 +317,7 @@ files:
 - lib/good_job/lockable.rb
 - lib/good_job/log_subscriber.rb
 - lib/good_job/multi_scheduler.rb
+- lib/good_job/notifier.rb
 - lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb