good_job 1.0.3 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fc7b3997aaa38583efb4c91e66a2a5f9b8bfd75d4c14c9baaf8f1af7c236a42e
-  data.tar.gz: 285b441a5e5add440142960884370a1b74168266252fa48ce569cbb56e80f240
+  metadata.gz: 7c6d495a4455453f6f4af736d3fd31685fc2026c98e0cd564ea0f47b188cc473
+  data.tar.gz: 794137e732ed3fcebec859daa8e58a2c2c8fd283cc43c5cfc7824263a1ca93fd
 SHA512:
-  metadata.gz: f54e83d6ac19d80da2123a395ccf6ffd7eb2faab8e68bd5d50a572c8f0bf64fbc907125a47b7ff91dccac5497af113d043bba4f62ee6ae10fecaf3c004a1114d
-  data.tar.gz: 0b65a60bde26db0cfcfa55f166e80043bcbccb6e2c80f5de1840d2deae3ee28880c904bb53ffc2478ad834ffef8cfe567cee9a179932e20e4f92fdf8504838e4
+  metadata.gz: 258757117262f25f5507ceb47c20f0f43a1f40427c7a0f9c3d0dc7803f25019062a160de08e96fd5ba74ceeeec5666d1084f6dff7ca9d381b2bbb198fabb10fd
+  data.tar.gz: e2a5d09cfb57a7f2a08d59f84c3b39e3582cb5b0b76a7e6ea2da421542d5e0ebee2e25a79298c1f5d3351d58e156e43ac0770e1d9e26e0f26169620f0bfcc493

data/CHANGELOG.md

@@ -1,6 +1,25 @@
 # Changelog
 
-## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-25)
+## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-09)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.3...v1.1.0)
+
+**Closed issues:**
+
+- Document reliability guarantees [\#59](https://github.com/bensheldon/good_job/issues/59)
+- Document how to hook in exception monitor \(Sentry, Rollbar, etc\) [\#47](https://github.com/bensheldon/good_job/issues/47)
+- Allow an Async mode [\#27](https://github.com/bensheldon/good_job/issues/27)
+
+**Merged pull requests:**
+
+- Add a callable hook on thread errors [\#71](https://github.com/bensheldon/good_job/pull/71) ([bensheldon](https://github.com/bensheldon))
+- Clarify reliability guarantees [\#70](https://github.com/bensheldon/good_job/pull/70) ([bensheldon](https://github.com/bensheldon))
+- Clean up Readme formatting; re-arrange tests for clarity and values [\#69](https://github.com/bensheldon/good_job/pull/69) ([bensheldon](https://github.com/bensheldon))
+- Create an Async execution mode [\#68](https://github.com/bensheldon/good_job/pull/68) ([bensheldon](https://github.com/bensheldon))
+- Move all stdout to LogSubscriber [\#67](https://github.com/bensheldon/good_job/pull/67) ([bensheldon](https://github.com/bensheldon))
+- Allow schedulers to be restarted; separate unit tests from integration tests [\#66](https://github.com/bensheldon/good_job/pull/66) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-26)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.2...v1.0.3)
 
@@ -113,7 +132,6 @@
 - Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
 - Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
 - Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
-- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
 
@@ -138,6 +156,7 @@
 
 **Merged pull requests:**
 
+- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
 - Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
 - Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
 - Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))

data/README.md

@@ -27,7 +27,8 @@ $ bundle install
 ## Usage
 
 1. Create a database migration:
-    ```bash
+    
+   ```bash
     $ bin/rails g good_job:install
     ```
 
@@ -38,7 +39,8 @@ $ bundle install
     ```
     
 1. Configure the ActiveJob adapter:
-    ```ruby
+    
+   ```ruby
     # config/application.rb
     config.active_job.queue_adapter = :good_job
     ```
@@ -57,16 +59,19 @@ $ bundle install
     ```
 
 1. Queue your job 🎉: 
+    
     ```ruby
     YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
+    
     ```bash
     $ bundle exec good_job
     ```
    
    Configuration options available with `help`:
+
    ```bash
    $ bundle exec good_job help start
    
@@ -81,7 +86,7 @@ $ bundle install
 
 ### Error handling, retries, and reliability
 
-GoodJob guarantees _at-least-once_ performance of jobs. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. 
+GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. Writing reliable, transactional, and idempotent `ActiveJob#perform` methods is outside the scope of GoodJob.
 
 #### Error handling
 
@@ -95,6 +100,15 @@ By default, if a job raises an error while it is being performed, _and it bubble
     GoodJob.reperform_jobs_on_standard_error = true # => default
     ```
 
+To report errors that _do_ bubble up to the GoodJob backend, assign a callable to `GoodJob.on_thread_error`. For example:
+
+```ruby
+# config/initializers/good_job.rb
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+```
+
 ### Retrying jobs
 
 ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
@@ -123,12 +137,29 @@ end
 
 GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
 
-    ```ruby
-    # config/initializers/good_job.rb
-    
-    # Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
-    GoodJob.reperform_jobs_on_standard_error = false 
-    ```
+```ruby
+# config/initializers/good_job.rb
+
+# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
+GoodJob.reperform_jobs_on_standard_error = false 
+```
+
+When using an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), the use of `rescue_on` may be incompatible with their ActiveJob integration. It's safest to explicitly wrap jobs with an exception reporter. For example:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  
+  around_perform do |_job, block|
+    block.call
+  rescue StandardError => e
+    Raven.capture_exception(e)
+    raise
+  end
+  # ...
+end
+```
+
 
 ActiveJob's `discard_on` functionality is supported too.
 
@@ -139,6 +170,14 @@ Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::Del
 ```ruby
 # config/initializers/good_job.rb
 ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+ActionMailer::DeliveryJob.around_perform do |_job, block|
+  block.call
+rescue StandardError => e
+  Raven.capture_exception(e)
+  raise
+end
 ```
 
 #### Timeouts
@@ -158,7 +197,7 @@ class ApplicationJob < ActiveJob::Base
 end
 ```
 
-### Configuring Job Execution Threads
+### Configuring job execution threads
     
 GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
 
@@ -169,6 +208,51 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Executing jobs async / in-process
+
+GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
+
+- Directly configure the ActiveJob adapter:
+
+    ```ruby
+    # config/environments/production.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
+    ```
+- Or, when using `...queue_adapter = :good_job`, via environment variables:
+
+    ```bash
+    $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
+    ```
+ 
+Depending on your application configuration, you may need to take additional steps:
+
+- Ensure that you have enough database connections for both web and job execution threads:
+
+    ```yaml
+    # config/database.yml
+    pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
+    ```
+
+- When running Puma with workers (`WEB_CONCURRENCY > 0`) or another process-forking webserver, GoodJob's threadpool schedulers should be stopped before forking, restarted after fork, and cleanly shut down on exit. Stopping GoodJob's scheduler pre-fork is recommended to ensure that GoodJob does not continue executing jobs in the parent/controller process. For example, with Puma:
+
+    ```ruby
+    # config/puma.rb
+  
+    before_fork do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    
+    on_worker_boot do
+      GoodJob::Scheduler.instances.each { |s| s.restart }
+    end
+    
+    on_worker_shutdown do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    ```
+  
+  GoodJob is compatible with Puma's `preload_app!` method.
+  
 ### Migrating to GoodJob from a different ActiveJob backend
 
 If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
@@ -184,7 +268,8 @@ If your application is already using an ActiveJob backend, you will need to inst
     ```
 
 1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
-    ```procfile
+    
+   ```procfile
     # Procfile
     # ...
     worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
@@ -210,15 +295,24 @@ It is also necessary to delete these preserved jobs from the database after a ce
 - For example, in a Rake task:
   
     ```ruby
-    # GoodJob::Job.finished(1.day.ago).delete_all
+    GoodJob::Job.finished(1.day.ago).delete_all
     ```
+
 - For example, using the `good_job` command-line utility:
 
     ```bash
     $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
     ```
 
-## Development
+## Contributing
+
+Contributions are welcomed and appreciated 🙏
+
+- Review the [Prioritized Project Backlog](https://github.com/bensheldon/good_job/projects/1).
+- Open a new Issue or contribute to an [existing Issue](https://github.com/bensheldon/good_job/issues). Questions or suggestions are fantastic.
+- Participate according to our [Code of Conduct](https://github.com/bensheldon/good_job/projects/1).
+
+### Gem development
 
 To run tests:
 
@@ -241,7 +335,6 @@ $ bundle exec appraisal
 
 # Run tests
 $ bundle exec appraisal bin/rspec
-
 ```
 
 For developing locally within another Ruby on Rails project:
@@ -256,24 +349,23 @@ $ bundle install
 # => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
 ```
 
-## Releasing
+### Releasing
 
-Package maintainers can release this gem with the following [gem-release](https://github.com/svenfuchs/gem-release) command:
+Package maintainers can release this gem by running:
 
 ```bash
 # Sign into rubygems
 $ gem signin
 
+# Add a .env file with the following:
+# CHANGELOG_GITHUB_TOKEN= # Github Personal Access Token
+
 # Update version number, changelog, and create git commit:
-$ bundle exec rake commit_version[minor] # major,minor,patch
+$ bundle exec rake release[minor] # major,minor,patch
 
 # ..and follow subsequent directions. 
 ```
 
-## Contributing
-
-Contribution directions go here.
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/active_job/queue_adapters/good_job_adapter.rb

@@ -1,18 +1,35 @@
 module ActiveJob
   module QueueAdapters
     class GoodJobAdapter < GoodJob::Adapter
-      def initialize(execution_mode: nil)
+      def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil)
         execution_mode = if execution_mode
                            execution_mode
                          elsif ENV['GOOD_JOB_EXECUTION_MODE'].present?
                            ENV['GOOD_JOB_EXECUTION_MODE'].to_sym
-                         elsif Rails.env.development? || Rails.env.test?
+                         elsif Rails.env.development?
+                           :inline
+                         elsif Rails.env.test?
                            :inline
                          else
                            :external
                          end
 
-        super(execution_mode: execution_mode)
+        if execution_mode == :async && scheduler.blank?
+          max_threads = (
+            max_threads.presence ||
+            ENV['GOOD_JOB_MAX_THREADS'] ||
+            ENV['RAILS_MAX_THREADS'] ||
+            ActiveRecord::Base.connection_pool.size
+          ).to_i
+
+          poll_interval = (
+            poll_interval.presence ||
+            ENV['GOOD_JOB_POLL_INTERVAL'] ||
+            1
+          ).to_i
+        end
+
+        super(execution_mode: execution_mode, max_threads: max_threads, poll_interval: poll_interval, scheduler: scheduler)
       end
     end
   end

data/lib/good_job.rb

@@ -1,7 +1,7 @@
 require "rails"
 require 'good_job/railtie'
 
-require 'good_job/logging'
+require 'good_job/log_subscriber'
 require 'good_job/lockable'
 require 'good_job/job'
 require 'good_job/scheduler'
@@ -12,9 +12,10 @@ require 'good_job/performer'
 require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
+  cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
   mattr_accessor :preserve_job_records, default: false
   mattr_accessor :reperform_jobs_on_standard_error, default: true
-  include Logging
+  mattr_accessor :on_thread_error, default: nil
 
   ActiveSupport.run_load_hooks(:good_job, self)
 end

data/lib/good_job/adapter.rb

@@ -1,8 +1,8 @@
 module GoodJob
   class Adapter
-    EXECUTION_MODES = [:inline, :external].freeze # TODO: async
+    EXECUTION_MODES = [:async, :external, :inline].freeze
 
-    def initialize(execution_mode: nil, inline: false)
+    def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
       if inline
         ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
         @execution_mode = :inline
@@ -13,6 +13,18 @@ module GoodJob
       else
         @execution_mode = :external
       end
+
+      @scheduler = scheduler
+      if @execution_mode == :async && @scheduler.blank? # rubocop:disable Style/GuardClause
+        timer_options = {}
+        timer_options[:execution_interval] = poll_interval if poll_interval.present?
+
+        pool_options = {}
+        pool_options[:max_threads] = max_threads if max_threads.present?
+
+        job_performer = GoodJob::Performer.new(GoodJob::Job, :perform_with_advisory_lock, name: '*')
+        @scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
+      end
     end
 
     def enqueue(active_job)
@@ -34,11 +46,21 @@ module GoodJob
         end
       end
 
+      @scheduler.create_thread if execute_async?
+
       good_job
     end
 
-    def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
-      nil
+    def shutdown(wait: true)
+      @scheduler&.shutdown(wait: wait)
+    end
+
+    def execute_async?
+      @execution_mode == :async
+    end
+
+    def execute_externally?
+      @execution_mode == :external
     end
 
     def execute_inline?
@@ -49,9 +71,5 @@ module GoodJob
       ActiveSupport::Deprecation.warn('GoodJob::Adapter::inline? is deprecated; use GoodJob::Adapter::execute_inline? instead')
       execute_inline?
     end
-
-    def execute_externally?
-      @execution_mode == :external
-    end
   end
 end

data/lib/good_job/cli.rb

@@ -16,7 +16,7 @@ module GoodJob
                   type: :numeric,
                   desc: "Interval between polls for available jobs in seconds (default: 1)"
     def start
-      require RAILS_ENVIRONMENT_RB
+      set_up_application!
 
       max_threads = (
         options[:max_threads] ||
@@ -25,23 +25,19 @@ module GoodJob
         ActiveRecord::Base.connection_pool.size
       ).to_i
 
-      queue_names = (
+      queue_string = (
         options[:queues] ||
         ENV['GOOD_JOB_QUEUES'] ||
         '*'
-      ).split(',').map(&:strip)
+      )
 
       poll_interval = (
         options[:poll_interval] ||
         ENV['GOOD_JOB_POLL_INTERVAL']
       ).to_i
 
-      job_query = GoodJob::Job.all.priority_ordered
-      queue_names_without_all = queue_names.reject { |q| q == '*' }
-      job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
-      job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock)
-
-      $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
+      job_query = GoodJob::Job.queue_string(queue_string)
+      job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
 
       timer_options = {}
       timer_options[:execution_interval] = poll_interval if poll_interval.positive?
@@ -62,24 +58,31 @@ module GoodJob
         break if @stop_good_job_executable || scheduler.shutdown?
       end
 
-      $stdout.puts "\nFinishing GoodJob's current jobs before exiting..."
       scheduler.shutdown
-      $stdout.puts "GoodJob's jobs finished, exiting..."
     end
 
+    default_task :start
+
     desc :cleanup_preserved_jobs, "Delete preserved job records"
     method_option :before_seconds_ago,
                   type: :numeric,
                   default: 24 * 60 * 60,
                   desc: "Delete records finished more than this many seconds ago"
     def cleanup_preserved_jobs
-      require RAILS_ENVIRONMENT_RB
+      set_up_application!
 
       timestamp = Time.current - options[:before_seconds_ago]
-      result = GoodJob::Job.finished(timestamp).delete_all
-      $stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
+      ActiveSupport::Notifications.instrument("cleanup_preserved_jobs.good_job", { before_seconds_ago: options[:before_seconds_ago], timestamp: timestamp }) do |payload|
+        deleted_records_count = GoodJob::Job.finished(timestamp).delete_all
+
+        payload[:deleted_records_count] = deleted_records_count
+      end
     end
 
-    default_task :start
+    no_commands do
+      def set_up_application!
+        require RAILS_ENVIRONMENT_RB
+      end
+    end
   end
 end

data/lib/good_job/job.rb

@@ -18,15 +18,19 @@ module GoodJob
       end
     end)
     scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
-    scope :priority_ordered, -> { order(priority: :desc) }
+    scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
     scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
+    scope :queue_string, (lambda do |string|
+      queue_names_without_all = (string.presence || '*').split(',').map(&:strip).reject { |q| q == '*' }
+      where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
+    end)
 
     def self.perform_with_advisory_lock
       good_job = nil
       result = nil
       error = nil
 
-      unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
+      unfinished.priority_ordered.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
         good_job = good_jobs.first
         break unless good_job
 

data/lib/good_job/log_subscriber.rb

@@ -0,0 +1,110 @@
+module GoodJob
+  class LogSubscriber < ActiveSupport::LogSubscriber
+    def create(event)
+      good_job = event.payload[:good_job]
+
+      debug do
+        "GoodJob created job resource with id #{good_job.id}"
+      end
+    end
+
+    def timer_task_finished(event)
+      exception = event.payload[:error]
+      return unless exception
+
+      error do
+        "GoodJob error: #{exception}\n #{exception.backtrace}"
+      end
+    end
+
+    def job_finished(event)
+      exception = event.payload[:error]
+      return unless exception
+
+      error do
+        "GoodJob error: #{exception}\n #{exception.backtrace}"
+      end
+    end
+
+    def scheduler_create_pools(event)
+      max_threads = event.payload[:max_threads]
+      poll_interval = event.payload[:poll_interval]
+      performer_name = event.payload[:performer_name]
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob started scheduler with queues=#{performer_name} max_threads=#{max_threads} poll_interval=#{poll_interval}."
+      end
+    end
+
+    def scheduler_shutdown_start(event)
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob shutting down scheduler..."
+      end
+    end
+
+    def scheduler_shutdown(event)
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob scheduler is shut down."
+      end
+    end
+
+    def scheduler_restart_pools(event)
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob scheduler has restarted."
+      end
+    end
+
+    def cleanup_preserved_jobs(event)
+      timestamp = event.payload[:timestamp]
+      deleted_records_count = event.payload[:deleted_records_count]
+
+      info_and_stdout do
+        "GoodJob deleted #{deleted_records_count} preserved #{'job'.pluralize(deleted_records_count)} finished before #{timestamp}."
+      end
+    end
+
+    private
+
+    def logger
+      GoodJob.logger
+    end
+
+    %w(info debug warn error fatal unknown).each do |level|
+      class_eval <<-METHOD, __FILE__, __LINE__ + 1
+        def #{level}(progname = nil, tags: [], &block)
+          return unless logger
+
+          if logger.respond_to?(:tagged)
+            tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
+            logger.tagged(*tags.compact) do
+              logger.#{level}(progname, &block)
+            end
+          else
+            logger.#{level}(progname, &block)
+          end
+        end
+      METHOD
+    end
+
+    def info_and_stdout(progname = nil, tags: [], &block)
+      unless ActiveSupport::Logger.logger_outputs_to?(logger, STDOUT)
+        tags_string = (['GoodJob'] + tags).map { |t| "[#{t}]" }.join(' ')
+        stdout_message = "#{tags_string}#{yield}"
+        $stdout.puts stdout_message
+      end
+
+      info(progname, tags: [], &block)
+    end
+
+    def thread_name
+      Thread.current.name || Thread.current.object_id
+    end
+  end
+end

data/lib/good_job/performer.rb

@@ -1,8 +1,11 @@
 module GoodJob
   class Performer
-    def initialize(target, method_name)
+    attr_reader :name
+
+    def initialize(target, method_name, name: nil)
       @target = target
       @method_name = method_name
+      @name = name
     end
 
     def next

data/lib/good_job/railtie.rb

@@ -2,6 +2,7 @@ module GoodJob
   class Railtie < ::Rails::Railtie
     initializer "good_job.logger" do
       ActiveSupport.on_load(:good_job) { self.logger = ::Rails.logger }
+      GoodJob::LogSubscriber.attach_to :good_job
     end
   end
 end

data/lib/good_job/scheduler.rb

@@ -20,32 +20,31 @@ module GoodJob
       fallback_policy: :discard,
     }.freeze
 
+    cattr_reader :instances, default: [], instance_reader: false
+
     def initialize(performer, timer_options: {}, pool_options: {})
       raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 
+      self.class.instances << self
+
       @performer = performer
-      @pool = ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
-      @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
-        create_thread
-      end
-      @timer.add_observer(self, :timer_observer)
-      @timer.execute
-    end
+      @pool_options = DEFAULT_POOL_OPTIONS.merge(pool_options)
+      @timer_options = DEFAULT_TIMER_OPTIONS.merge(timer_options)
 
-    def execute
+      create_pools
     end
 
     def shutdown(wait: true)
       @_shutdown = true
 
-      ActiveSupport::Notifications.instrument("scheduler_start_shutdown.good_job", { wait: wait })
-      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait }) do
-        if @timer.running?
+      ActiveSupport::Notifications.instrument("scheduler_shutdown_start.good_job", { wait: wait, process_id: process_id })
+      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait, process_id: process_id }) do
+        if @timer&.running?
           @timer.shutdown
           @timer.wait_for_termination if wait
         end
 
-        if @pool.running?
+        if @pool&.running?
           @pool.shutdown
           @pool.wait_for_termination if wait
         end
@@ -56,6 +55,13 @@ module GoodJob
       @_shutdown
     end
 
+    def restart(wait: true)
+      ActiveSupport::Notifications.instrument("scheduler_restart_pools.good_job", { process_id: process_id }) do
+        shutdown(wait: wait) unless shutdown?
+        create_pools
+      end
+    end
+
     def create_thread
       return false unless @pool.ready_worker_count.positive?
 
@@ -69,10 +75,12 @@ module GoodJob
     end
 
     def timer_observer(time, executed_task, thread_error)
+      GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
       ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
     end
 
     def task_observer(time, output, thread_error)
+      GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
       ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
       create_thread if output
     end
@@ -88,5 +96,26 @@ module GoodJob
         end
       end
     end
+
+    private
+
+    def create_pools
+      ActiveSupport::Notifications.instrument("scheduler_create_pools.good_job", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval], process_id: process_id }) do
+        @pool = ThreadPoolExecutor.new(@pool_options)
+        next unless @timer_options[:execution_interval].positive?
+
+        @timer = Concurrent::TimerTask.new(@timer_options) { create_thread }
+        @timer.add_observer(self, :timer_observer)
+        @timer.execute
+      end
+    end
+
+    def process_id
+      Process.pid
+    end
+
+    def thread_name
+      Thread.current.name || Thread.current.object_id
+    end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '1.0.3'.freeze
+  VERSION = '1.1.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.0.3
+  version: 1.1.0
 platform: ruby
 authors:
 - Ben Sheldon
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-07-26 00:00:00.000000000 Z
+date: 2020-08-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -94,6 +94,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: dotenv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: foreman
   requirement: !ruby/object:Gem::Requirement
@@ -150,6 +164,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: puma
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
@@ -243,7 +271,7 @@ files:
 - lib/good_job/cli.rb
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
-- lib/good_job/logging.rb
+- lib/good_job/log_subscriber.rb
 - lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb
@@ -258,7 +286,7 @@ metadata:
   documentation_uri: https://rdoc.info/github/bensheldon/good_job
   homepage_uri: https://github.com/bensheldon/good_job
   source_code_uri: https://github.com/bensheldon/good_job
-post_install_message: 
+post_install_message:
 rdoc_options:
 - "--title"
 - GoodJob - a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
@@ -281,7 +309,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.0.3
-signing_key: 
+signing_key:
 specification_version: 4
 summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
 test_files: []

data/lib/good_job/logging.rb

@@ -1,70 +0,0 @@
-module GoodJob
-  module Logging
-    extend ActiveSupport::Concern
-
-    included do
-      cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
-
-      def self.tag_logger(*tags)
-        if logger.respond_to?(:tagged)
-          tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
-          logger.tagged(*tags) { yield }
-        else
-          yield
-        end
-      end
-    end
-
-    class LogSubscriber < ActiveSupport::LogSubscriber
-      def create(event)
-        good_job = event.payload[:good_job]
-
-        info do
-          "Created GoodJob resource with id #{good_job.id}"
-        end
-      end
-
-      def timer_task_finished(event)
-        exception = event.payload[:error]
-        return unless exception
-
-        error do
-          "ERROR: #{exception}\n #{exception.backtrace}"
-        end
-      end
-
-      def job_finished(event)
-        exception = event.payload[:error]
-        return unless exception
-
-        error do
-          "ERROR: #{exception}\n #{exception.backtrace}"
-        end
-      end
-
-      def scheduler_start_shutdown(_event)
-        info do
-          "Shutting down scheduler..."
-        end
-      end
-
-      def scheduler_shutdown(_event)
-        info do
-          "Scheduler is shut down."
-        end
-      end
-
-      private
-
-      def logger
-        GoodJob.logger
-      end
-
-      def thread_name
-        Thread.current.name || Thread.current.object_id
-      end
-    end
-  end
-end
-
-GoodJob::Logging::LogSubscriber.attach_to :good_job