good_job 0.9.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bddc60fa297293af01b1334d4adf51981bfb8b314ba4577a7b3bff1d7385bd5a
-  data.tar.gz: 3533f21ffee4dba606558fa0fadd1f37a943aa285b4f6bebed4bb3b89045b61e
+  metadata.gz: 7c6d495a4455453f6f4af736d3fd31685fc2026c98e0cd564ea0f47b188cc473
+  data.tar.gz: 794137e732ed3fcebec859daa8e58a2c2c8fd283cc43c5cfc7824263a1ca93fd
 SHA512:
-  metadata.gz: 74c0d3df2df08e78d06baa8982401d633af536277231a17d87687ff3c7ba1553d4123f7c1ce49ae23094bd781db5ee23eb9584eeba94aae49fb8cc83da0c68e0
-  data.tar.gz: 7df20f03621cb66b864a2c2950e4a11f60199cd4954ad5a622a1631254fa04bab1243ee376567600a2aed294a8c14fa9802fd1024855dc7d473a7ff8f7166515
+  metadata.gz: 258757117262f25f5507ceb47c20f0f43a1f40427c7a0f9c3d0dc7803f25019062a160de08e96fd5ba74ceeeec5666d1084f6dff7ca9d381b2bbb198fabb10fd
+  data.tar.gz: e2a5d09cfb57a7f2a08d59f84c3b39e3582cb5b0b76a7e6ea2da421542d5e0ebee2e25a79298c1f5d3351d58e156e43ac0770e1d9e26e0f26169620f0bfcc493

data/CHANGELOG.md

@@ -1,21 +1,117 @@
 # Changelog
 
+## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-09)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.3...v1.1.0)
+
+**Closed issues:**
+
+- Document reliability guarantees [\#59](https://github.com/bensheldon/good_job/issues/59)
+- Document how to hook in exception monitor \(Sentry, Rollbar, etc\) [\#47](https://github.com/bensheldon/good_job/issues/47)
+- Allow an Async mode [\#27](https://github.com/bensheldon/good_job/issues/27)
+
+**Merged pull requests:**
+
+- Add a callable hook on thread errors [\#71](https://github.com/bensheldon/good_job/pull/71) ([bensheldon](https://github.com/bensheldon))
+- Clarify reliability guarantees [\#70](https://github.com/bensheldon/good_job/pull/70) ([bensheldon](https://github.com/bensheldon))
+- Clean up Readme formatting; re-arrange tests for clarity and values [\#69](https://github.com/bensheldon/good_job/pull/69) ([bensheldon](https://github.com/bensheldon))
+- Create an Async execution mode [\#68](https://github.com/bensheldon/good_job/pull/68) ([bensheldon](https://github.com/bensheldon))
+- Move all stdout to LogSubscriber [\#67](https://github.com/bensheldon/good_job/pull/67) ([bensheldon](https://github.com/bensheldon))
+- Allow schedulers to be restarted; separate unit tests from integration tests [\#66](https://github.com/bensheldon/good_job/pull/66) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-26)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.2...v1.0.3)
+
+**Fixed bugs:**
+
+- Preserve GoodJob::Jobs when a StandardError is raised [\#60](https://github.com/bensheldon/good_job/issues/60)
+
+**Closed issues:**
+
+- Have an initial setup generator [\#6](https://github.com/bensheldon/good_job/issues/6)
+
+**Merged pull requests:**
+
+- Re-perform a job if a StandardError bubbles up; better document job reliability [\#62](https://github.com/bensheldon/good_job/pull/62) ([bensheldon](https://github.com/bensheldon))
+- Update the setup documentation to use correct bin setup command [\#61](https://github.com/bensheldon/good_job/pull/61) ([jm96441n](https://github.com/jm96441n))
+
+## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-25)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.1...v1.0.2)
+
+**Fixed bugs:**
+
+- Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- repeating/recurring jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
+
+**Merged pull requests:**
+
+- Add migration generator [\#56](https://github.com/bensheldon/good_job/pull/56) ([thedanbob](https://github.com/thedanbob))
+- Fix migration script in readme [\#55](https://github.com/bensheldon/good_job/pull/55) ([thedanbob](https://github.com/thedanbob))
+
+## [v1.0.1](https://github.com/bensheldon/good_job/tree/v1.0.1) (2020-07-22)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.0...v1.0.1)
+
+**Merged pull requests:**
+
+- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.9.0...v1.0.0)
+
 ## [v0.9.0](https://github.com/bensheldon/good_job/tree/v0.9.0) (2020-07-20)
 
-[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.9.0)
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.2...v0.9.0)
+
+**Merged pull requests:**
+
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.8.2](https://github.com/bensheldon/good_job/tree/v0.8.2) (2020-07-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.1...v0.8.2)
 
 **Closed issues:**
 
-- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
 - Add a job timeout configuration to time out jobs that have run too long [\#19](https://github.com/bensheldon/good_job/issues/19)
 
 **Merged pull requests:**
 
-- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
 - Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
 - Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
+
+## [v0.8.1](https://github.com/bensheldon/good_job/tree/v0.8.1) (2020-07-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.0...v0.8.1)
+
+**Merged pull requests:**
+
 - Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.8.0](https://github.com/bensheldon/good_job/tree/v0.8.0) (2020-07-17)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.7.0...v0.8.0)
+
+**Merged pull requests:**
+
 - Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-16)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.7.0)
+
+**Closed issues:**
+
+- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
+
+**Merged pull requests:**
+
 - Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
 - Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
 - Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))

data/README.md

@@ -9,6 +9,8 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 - **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
+For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -25,35 +27,11 @@ $ bundle install
 ## Usage
 
 1. Create a database migration:
-    ```bash
-    $ bin/rails g migration CreateGoodJobs
+    
+   ```bash
+    $ bin/rails g good_job:install
     ```
 
-    Add to the newly created migration file:
-
-    ```ruby
-    class CreateGoodJobs < ActiveRecord::Migration[6.0]
-      def change
-        enable_extension 'pgcrypto'
-
-        create_table :good_jobs, id: :uuid do |t|
-          t.timestamps
-
-          t.text :queue_name
-          t.integer :priority
-          t.jsonb :serialized_params
-          t.timestamp :scheduled_at
-          t.timestamp :performed_at
-          t.timestamp :finished_at
-          t.text :error
-
-          add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
-          add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
-        end
-      end
-    end
-    ```
-    
     Run the migration:
     
     ```bash
@@ -61,7 +39,8 @@ $ bundle install
     ```
     
 1. Configure the ActiveJob adapter:
-    ```ruby
+    
+   ```ruby
     # config/application.rb
     config.active_job.queue_adapter = :good_job
     ```
@@ -80,16 +59,19 @@ $ bundle install
     ```
 
 1. Queue your job 🎉: 
+    
     ```ruby
     YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
+    
     ```bash
     $ bundle exec good_job
     ```
    
    Configuration options available with `help`:
+
    ```bash
    $ bundle exec good_job help start
    
@@ -102,26 +84,120 @@ $ bundle install
    #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
    ```
 
-### Taking advantage of ActiveJob
+### Error handling, retries, and reliability
+
+GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. Writing reliable, transactional, and idempotent `ActiveJob#perform` methods is outside the scope of GoodJob.
+
+#### Error handling
+
+By default, if a job raises an error while it is being performed, _and it bubbles up to the GoodJob backend_, GoodJob will be immediately re-perform the job until it finishes successfully.
+
+- `Exception`-type errors, such as a SIGINT, will always cause a job to be re-performed.
+- `StandardError`-type errors, by default, will cause a job to be re-performed, though this is configurable:
+   
+    ```ruby
+    # config/initializers/good_job.rb
+    GoodJob.reperform_jobs_on_standard_error = true # => default
+    ```
+
+To report errors that _do_ bubble up to the GoodJob backend, assign a callable to `GoodJob.on_thread_error`. For example:
+
+```ruby
+# config/initializers/good_job.rb
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+```
+
+### Retrying jobs
+
+ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  # ...
+end
+```
+
+When specifying a limited number of retries, care must be taken to ensure that an error does not bubble up to the GoodJob backend because that will result in the job being re-performed:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, attempts: 5 do |_job, _exception|
+    # Log error, etc.
+    # You must implement this block, otherwise, 
+    #   Active Job will re-raise the error.
+    # Do not re-raise the error, otherwise 
+    #   GoodJob will immediately re-perform the job. 
+  end
+  # ...
+end
+```
 
-ActiveJob has a rich set of built-in functionality for timeouts, error handling, and retrying. For example:
+GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
+
+```ruby
+# config/initializers/good_job.rb
+
+# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
+GoodJob.reperform_jobs_on_standard_error = false 
+```
+
+When using an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), the use of `rescue_on` may be incompatible with their ActiveJob integration. It's safest to explicitly wrap jobs with an exception reporter. For example:
 
 ```ruby
 class ApplicationJob < ActiveJob::Base  
-  # Retry errors an infinite number of times with exponential back-off
   retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  
+  around_perform do |_job, block|
+    block.call
+  rescue StandardError => e
+    Raven.capture_exception(e)
+    raise
+  end
+  # ...
+end
+```
+
+
+ActiveJob's `discard_on` functionality is supported too.
+
+#### ActionMailer retries
 
-  # Timeout jobs after 10 minutes
+Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::DeliveryJob` which inherits from `ActiveJob::Base` rather than your applications `ApplicationJob`. You can use an initializer to configure retries on `ActionMailer::DeliveryJob`:
+
+```ruby
+# config/initializers/good_job.rb
+ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+ActionMailer::DeliveryJob.around_perform do |_job, block|
+  block.call
+rescue StandardError => e
+  Raven.capture_exception(e)
+  raise
+end
+```
+
+#### Timeouts
+
+Job timeouts can be configured with an `around_perform`:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
   JobTimeoutError = Class.new(StandardError)
+  
   around_perform do |_job, block|
+    # Timeout jobs after 10 minutes
     Timeout.timeout(10.minutes, JobTimeoutError) do
       block.call
     end
   end
 end
 ```
-   
-### Configuring Job Execution Threads
+
+### Configuring job execution threads
     
 GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
 
@@ -132,6 +208,51 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Executing jobs async / in-process
+
+GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
+
+- Directly configure the ActiveJob adapter:
+
+    ```ruby
+    # config/environments/production.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
+    ```
+- Or, when using `...queue_adapter = :good_job`, via environment variables:
+
+    ```bash
+    $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
+    ```
+ 
+Depending on your application configuration, you may need to take additional steps:
+
+- Ensure that you have enough database connections for both web and job execution threads:
+
+    ```yaml
+    # config/database.yml
+    pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
+    ```
+
+- When running Puma with workers (`WEB_CONCURRENCY > 0`) or another process-forking webserver, GoodJob's threadpool schedulers should be stopped before forking, restarted after fork, and cleanly shut down on exit. Stopping GoodJob's scheduler pre-fork is recommended to ensure that GoodJob does not continue executing jobs in the parent/controller process. For example, with Puma:
+
+    ```ruby
+    # config/puma.rb
+  
+    before_fork do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    
+    on_worker_boot do
+      GoodJob::Scheduler.instances.each { |s| s.restart }
+    end
+    
+    on_worker_shutdown do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    ```
+  
+  GoodJob is compatible with Puma's `preload_app!` method.
+  
 ### Migrating to GoodJob from a different ActiveJob backend
 
 If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
@@ -147,7 +268,8 @@ If your application is already using an ActiveJob backend, you will need to inst
     ```
 
 1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
-    ```procfile
+    
+   ```procfile
     # Procfile
     # ...
     worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
@@ -173,15 +295,24 @@ It is also necessary to delete these preserved jobs from the database after a ce
 - For example, in a Rake task:
   
     ```ruby
-    # GoodJob::Job.finished(1.day.ago).delete_all
+    GoodJob::Job.finished(1.day.ago).delete_all
     ```
+
 - For example, using the `good_job` command-line utility:
 
     ```bash
     $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
     ```
 
-## Development
+## Contributing
+
+Contributions are welcomed and appreciated 🙏
+
+- Review the [Prioritized Project Backlog](https://github.com/bensheldon/good_job/projects/1).
+- Open a new Issue or contribute to an [existing Issue](https://github.com/bensheldon/good_job/issues). Questions or suggestions are fantastic.
+- Participate according to our [Code of Conduct](https://github.com/bensheldon/good_job/projects/1).
+
+### Gem development
 
 To run tests:
 
@@ -190,7 +321,7 @@ To run tests:
 $ git clone git@github.com:bensheldon/good_job.git
 
 # Set up the local environment
-$ bin/setup_test
+$ bin/setup
 
 # Run the tests
 $ bin/rspec
@@ -204,7 +335,6 @@ $ bundle exec appraisal
 
 # Run tests
 $ bundle exec appraisal bin/rspec
-
 ```
 
 For developing locally within another Ruby on Rails project:
@@ -219,24 +349,23 @@ $ bundle install
 # => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
 ```
 
-## Releasing
+### Releasing
 
-Package maintainers can release this gem with the following [gem-release](https://github.com/svenfuchs/gem-release) command:
+Package maintainers can release this gem by running:
 
 ```bash
 # Sign into rubygems
 $ gem signin
 
+# Add a .env file with the following:
+# CHANGELOG_GITHUB_TOKEN= # Github Personal Access Token
+
 # Update version number, changelog, and create git commit:
-$ bundle exec rake commit_version[minor] # major,minor,patch
+$ bundle exec rake release[minor] # major,minor,patch
 
 # ..and follow subsequent directions. 
 ```
 
-## Contributing
-
-Contribution directions go here.
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/active_job/queue_adapters/good_job_adapter.rb

@@ -1,18 +1,35 @@
 module ActiveJob
   module QueueAdapters
     class GoodJobAdapter < GoodJob::Adapter
-      def initialize(execution_mode: nil)
+      def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil)
         execution_mode = if execution_mode
                            execution_mode
                          elsif ENV['GOOD_JOB_EXECUTION_MODE'].present?
                            ENV['GOOD_JOB_EXECUTION_MODE'].to_sym
-                         elsif Rails.env.development? || Rails.env.test?
+                         elsif Rails.env.development?
+                           :inline
+                         elsif Rails.env.test?
                            :inline
                          else
                            :external
                          end
 
-        super(execution_mode: execution_mode)
+        if execution_mode == :async && scheduler.blank?
+          max_threads = (
+            max_threads.presence ||
+            ENV['GOOD_JOB_MAX_THREADS'] ||
+            ENV['RAILS_MAX_THREADS'] ||
+            ActiveRecord::Base.connection_pool.size
+          ).to_i
+
+          poll_interval = (
+            poll_interval.presence ||
+            ENV['GOOD_JOB_POLL_INTERVAL'] ||
+            1
+          ).to_i
+        end
+
+        super(execution_mode: execution_mode, max_threads: max_threads, poll_interval: poll_interval, scheduler: scheduler)
       end
     end
   end

data/lib/generators/good_job/install_generator.rb

@@ -0,0 +1,24 @@
+require 'rails/generators'
+require 'rails/generators/active_record'
+
+module GoodJob
+  class InstallGenerator < Rails::Generators::Base
+    include Rails::Generators::Migration
+
+    class << self
+      delegate :next_migration_number, to: ActiveRecord::Generators::Base
+    end
+
+    source_paths << File.join(File.dirname(__FILE__), "templates")
+
+    def create_migration_file
+      migration_template 'migration.rb.erb', 'db/migrate/create_good_jobs.rb', migration_version: migration_version
+    end
+
+    private
+
+    def migration_version
+      "[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]"
+    end
+  end
+end

data/lib/generators/good_job/templates/migration.rb.erb

@@ -0,0 +1,20 @@
+class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
+  def change
+    enable_extension 'pgcrypto'
+
+    create_table :good_jobs, id: :uuid do |t|
+      t.text :queue_name
+      t.integer :priority
+      t.jsonb :serialized_params
+      t.timestamp :scheduled_at
+      t.timestamp :performed_at
+      t.timestamp :finished_at
+      t.text :error
+
+      t.timestamps
+    end
+
+    add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
+    add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
+  end
+end

data/lib/good_job.rb

@@ -1,7 +1,7 @@
 require "rails"
 require 'good_job/railtie'
 
-require 'good_job/logging'
+require 'good_job/log_subscriber'
 require 'good_job/lockable'
 require 'good_job/job'
 require 'good_job/scheduler'
@@ -12,8 +12,10 @@ require 'good_job/performer'
 require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
+  cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
   mattr_accessor :preserve_job_records, default: false
-  include Logging
+  mattr_accessor :reperform_jobs_on_standard_error, default: true
+  mattr_accessor :on_thread_error, default: nil
 
   ActiveSupport.run_load_hooks(:good_job, self)
 end

data/lib/good_job/adapter.rb

@@ -1,8 +1,8 @@
 module GoodJob
   class Adapter
-    EXECUTION_MODES = [:inline, :external].freeze # TODO: async
+    EXECUTION_MODES = [:async, :external, :inline].freeze
 
-    def initialize(execution_mode: nil, inline: false)
+    def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
       if inline
         ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
         @execution_mode = :inline
@@ -13,6 +13,18 @@ module GoodJob
       else
         @execution_mode = :external
       end
+
+      @scheduler = scheduler
+      if @execution_mode == :async && @scheduler.blank? # rubocop:disable Style/GuardClause
+        timer_options = {}
+        timer_options[:execution_interval] = poll_interval if poll_interval.present?
+
+        pool_options = {}
+        pool_options[:max_threads] = max_threads if max_threads.present?
+
+        job_performer = GoodJob::Performer.new(GoodJob::Job, :perform_with_advisory_lock, name: '*')
+        @scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
+      end
     end
 
     def enqueue(active_job)
@@ -34,11 +46,21 @@ module GoodJob
         end
       end
 
+      @scheduler.create_thread if execute_async?
+
       good_job
     end
 
-    def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
-      nil
+    def shutdown(wait: true)
+      @scheduler&.shutdown(wait: wait)
+    end
+
+    def execute_async?
+      @execution_mode == :async
+    end
+
+    def execute_externally?
+      @execution_mode == :external
     end
 
     def execute_inline?
@@ -49,9 +71,5 @@ module GoodJob
       ActiveSupport::Deprecation.warn('GoodJob::Adapter::inline? is deprecated; use GoodJob::Adapter::execute_inline? instead')
       execute_inline?
     end
-
-    def execute_externally?
-      @execution_mode == :external
-    end
   end
 end

data/lib/good_job/cli.rb

@@ -16,7 +16,7 @@ module GoodJob
                   type: :numeric,
                   desc: "Interval between polls for available jobs in seconds (default: 1)"
     def start
-      require RAILS_ENVIRONMENT_RB
+      set_up_application!
 
       max_threads = (
         options[:max_threads] ||
@@ -25,29 +25,19 @@ module GoodJob
         ActiveRecord::Base.connection_pool.size
       ).to_i
 
-      queue_names = (
+      queue_string = (
         options[:queues] ||
         ENV['GOOD_JOB_QUEUES'] ||
         '*'
-      ).split(',').map(&:strip)
+      )
 
       poll_interval = (
         options[:poll_interval] ||
         ENV['GOOD_JOB_POLL_INTERVAL']
       ).to_i
 
-      job_query = GoodJob::Job.all.priority_ordered
-      queue_names_without_all = queue_names.reject { |q| q == '*' }
-      job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
-
-      performer_method = if GoodJob.preserve_job_records
-                           :perform_with_advisory_lock_and_preserve_job_records
-                         else
-                           :perform_with_advisory_lock_and_destroy_job_records
-                         end
-      job_performer = GoodJob::Performer.new(job_query, performer_method)
-
-      $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
+      job_query = GoodJob::Job.queue_string(queue_string)
+      job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
 
       timer_options = {}
       timer_options[:execution_interval] = poll_interval if poll_interval.positive?
@@ -68,24 +58,31 @@ module GoodJob
         break if @stop_good_job_executable || scheduler.shutdown?
       end
 
-      $stdout.puts "\nFinishing GoodJob's current jobs before exiting..."
       scheduler.shutdown
-      $stdout.puts "GoodJob's jobs finished, exiting..."
     end
 
+    default_task :start
+
     desc :cleanup_preserved_jobs, "Delete preserved job records"
     method_option :before_seconds_ago,
                   type: :numeric,
                   default: 24 * 60 * 60,
                   desc: "Delete records finished more than this many seconds ago"
     def cleanup_preserved_jobs
-      require RAILS_ENVIRONMENT_RB
+      set_up_application!
 
       timestamp = Time.current - options[:before_seconds_ago]
-      result = GoodJob::Job.finished(timestamp).delete_all
-      $stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
+      ActiveSupport::Notifications.instrument("cleanup_preserved_jobs.good_job", { before_seconds_ago: options[:before_seconds_ago], timestamp: timestamp }) do |payload|
+        deleted_records_count = GoodJob::Job.finished(timestamp).delete_all
+
+        payload[:deleted_records_count] = deleted_records_count
+      end
     end
 
-    default_task :start
+    no_commands do
+      def set_up_application!
+        require RAILS_ENVIRONMENT_RB
+      end
+    end
   end
 end

data/lib/good_job/job.rb

@@ -18,32 +18,28 @@ module GoodJob
       end
     end)
     scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
-    scope :priority_ordered, -> { order(priority: :desc) }
+    scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
     scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
+    scope :queue_string, (lambda do |string|
+      queue_names_without_all = (string.presence || '*').split(',').map(&:strip).reject { |q| q == '*' }
+      where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
+    end)
 
-    def self.perform_with_advisory_lock(destroy_after: !GoodJob.preserve_job_records)
+    def self.perform_with_advisory_lock
       good_job = nil
       result = nil
       error = nil
 
-      unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
+      unfinished.priority_ordered.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
         good_job = good_jobs.first
         break unless good_job
 
-        result, error = good_job.perform(destroy_after: destroy_after)
+        result, error = good_job.perform
       end
 
       [good_job, result, error] if good_job
     end
 
-    def self.perform_with_advisory_lock_and_preserve_job_records
-      perform_with_advisory_lock(destroy_after: false)
-    end
-
-    def self.perform_with_advisory_lock_and_destroy_job_records
-      perform_with_advisory_lock(destroy_after: true)
-    end
-
     def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
       good_job = nil
       ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
@@ -64,40 +60,49 @@ module GoodJob
       good_job
     end
 
-    def perform(destroy_after: true)
+    def perform(destroy_after: !GoodJob.preserve_job_records, reperform_on_standard_error: GoodJob.reperform_jobs_on_standard_error)
       raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
 
       result = nil
+      rescued_error = nil
       error = nil
 
       ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
       self.performed_at = Time.current
       save! unless destroy_after
 
-      ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
-        params = serialized_params.merge(
-          "provider_job_id" => id
-        )
-        begin
+      params = serialized_params.merge(
+        "provider_job_id" => id
+      )
+
+      begin
+        ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
           result = ActiveJob::Base.execute(params)
-        rescue StandardError => e
-          error = e
         end
+      rescue StandardError => e
+        rescued_error = e
       end
 
-      if error.nil? && result.is_a?(Exception)
+      if rescued_error
+        error = rescued_error
+      elsif result.is_a?(Exception)
         error = result
         result = nil
       end
 
       error_message = "#{error.class}: #{error.message}" if error
       self.error = error_message
-      self.finished_at = Time.current
 
-      if destroy_after
-        destroy!
-      else
+      if rescued_error && reperform_on_standard_error
         save!
+      else
+        self.finished_at = Time.current
+
+        if destroy_after
+          destroy!
+        else
+          save!
+        end
       end
 
       [result, error]

data/lib/good_job/log_subscriber.rb

@@ -0,0 +1,110 @@
+module GoodJob
+  class LogSubscriber < ActiveSupport::LogSubscriber
+    def create(event)
+      good_job = event.payload[:good_job]
+
+      debug do
+        "GoodJob created job resource with id #{good_job.id}"
+      end
+    end
+
+    def timer_task_finished(event)
+      exception = event.payload[:error]
+      return unless exception
+
+      error do
+        "GoodJob error: #{exception}\n #{exception.backtrace}"
+      end
+    end
+
+    def job_finished(event)
+      exception = event.payload[:error]
+      return unless exception
+
+      error do
+        "GoodJob error: #{exception}\n #{exception.backtrace}"
+      end
+    end
+
+    def scheduler_create_pools(event)
+      max_threads = event.payload[:max_threads]
+      poll_interval = event.payload[:poll_interval]
+      performer_name = event.payload[:performer_name]
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob started scheduler with queues=#{performer_name} max_threads=#{max_threads} poll_interval=#{poll_interval}."
+      end
+    end
+
+    def scheduler_shutdown_start(event)
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob shutting down scheduler..."
+      end
+    end
+
+    def scheduler_shutdown(event)
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob scheduler is shut down."
+      end
+    end
+
+    def scheduler_restart_pools(event)
+      process_id = event.payload[:process_id]
+
+      info_and_stdout(tags: [process_id]) do
+        "GoodJob scheduler has restarted."
+      end
+    end
+
+    def cleanup_preserved_jobs(event)
+      timestamp = event.payload[:timestamp]
+      deleted_records_count = event.payload[:deleted_records_count]
+
+      info_and_stdout do
+        "GoodJob deleted #{deleted_records_count} preserved #{'job'.pluralize(deleted_records_count)} finished before #{timestamp}."
+      end
+    end
+
+    private
+
+    def logger
+      GoodJob.logger
+    end
+
+    %w(info debug warn error fatal unknown).each do |level|
+      class_eval <<-METHOD, __FILE__, __LINE__ + 1
+        def #{level}(progname = nil, tags: [], &block)
+          return unless logger
+
+          if logger.respond_to?(:tagged)
+            tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
+            logger.tagged(*tags.compact) do
+              logger.#{level}(progname, &block)
+            end
+          else
+            logger.#{level}(progname, &block)
+          end
+        end
+      METHOD
+    end
+
+    def info_and_stdout(progname = nil, tags: [], &block)
+      unless ActiveSupport::Logger.logger_outputs_to?(logger, STDOUT)
+        tags_string = (['GoodJob'] + tags).map { |t| "[#{t}]" }.join(' ')
+        stdout_message = "#{tags_string}#{yield}"
+        $stdout.puts stdout_message
+      end
+
+      info(progname, tags: [], &block)
+    end
+
+    def thread_name
+      Thread.current.name || Thread.current.object_id
+    end
+  end
+end

data/lib/good_job/performer.rb

@@ -1,8 +1,11 @@
 module GoodJob
   class Performer
-    def initialize(target, method_name)
+    attr_reader :name
+
+    def initialize(target, method_name, name: nil)
       @target = target
       @method_name = method_name
+      @name = name
     end
 
     def next

data/lib/good_job/railtie.rb

@@ -2,6 +2,7 @@ module GoodJob
   class Railtie < ::Rails::Railtie
     initializer "good_job.logger" do
       ActiveSupport.on_load(:good_job) { self.logger = ::Rails.logger }
+      GoodJob::LogSubscriber.attach_to :good_job
     end
   end
 end

data/lib/good_job/scheduler.rb

@@ -15,38 +15,36 @@ module GoodJob
       min_threads: 0,
       max_threads: Concurrent.processor_count,
       auto_terminate: true,
-      idletime: 0,
-      max_queue: 0,
-      fallback_policy: :abort, # shouldn't matter -- 0 max queue
+      idletime: 60,
+      max_queue: -1,
+      fallback_policy: :discard,
     }.freeze
 
+    cattr_reader :instances, default: [], instance_reader: false
+
     def initialize(performer, timer_options: {}, pool_options: {})
       raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 
+      self.class.instances << self
+
       @performer = performer
-      @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
-      @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
-        idle_threads = @pool.max_length - @pool.length
-        create_thread if idle_threads.positive?
-      end
-      @timer.add_observer(self, :timer_observer)
-      @timer.execute
-    end
+      @pool_options = DEFAULT_POOL_OPTIONS.merge(pool_options)
+      @timer_options = DEFAULT_TIMER_OPTIONS.merge(timer_options)
 
-    def execute
+      create_pools
     end
 
     def shutdown(wait: true)
       @_shutdown = true
 
-      ActiveSupport::Notifications.instrument("scheduler_start_shutdown.good_job", { wait: wait })
-      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait }) do
-        if @timer.running?
+      ActiveSupport::Notifications.instrument("scheduler_shutdown_start.good_job", { wait: wait, process_id: process_id })
+      ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait, process_id: process_id }) do
+        if @timer&.running?
           @timer.shutdown
           @timer.wait_for_termination if wait
         end
 
-        if @pool.running?
+        if @pool&.running?
           @pool.shutdown
           @pool.wait_for_termination if wait
         end
@@ -57,7 +55,16 @@ module GoodJob
       @_shutdown
     end
 
+    def restart(wait: true)
+      ActiveSupport::Notifications.instrument("scheduler_restart_pools.good_job", { process_id: process_id }) do
+        shutdown(wait: wait) unless shutdown?
+        create_pools
+      end
+    end
+
     def create_thread
+      return false unless @pool.ready_worker_count.positive?
+
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
         output = nil
         Rails.application.executor.wrap { output = performer.next }
@@ -68,12 +75,47 @@ module GoodJob
     end
 
     def timer_observer(time, executed_task, thread_error)
+      GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
       ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
     end
 
     def task_observer(time, output, thread_error)
+      GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
       ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
       create_thread if output
     end
+
+    class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
+      # https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
+      def ready_worker_count
+        synchronize do
+          workers_still_to_be_created = @max_length - @pool.length
+          workers_created_but_waiting = @ready.length
+
+          workers_still_to_be_created + workers_created_but_waiting
+        end
+      end
+    end
+
+    private
+
+    def create_pools
+      ActiveSupport::Notifications.instrument("scheduler_create_pools.good_job", { performer_name: @performer.name, max_threads: @pool_options[:max_threads], poll_interval: @timer_options[:execution_interval], process_id: process_id }) do
+        @pool = ThreadPoolExecutor.new(@pool_options)
+        next unless @timer_options[:execution_interval].positive?
+
+        @timer = Concurrent::TimerTask.new(@timer_options) { create_thread }
+        @timer.add_observer(self, :timer_observer)
+        @timer.execute
+      end
+    end
+
+    def process_id
+      Process.pid
+    end
+
+    def thread_name
+      Thread.current.name || Thread.current.object_id
+    end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '0.9.0'.freeze
+  VERSION = '1.1.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Ben Sheldon
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-07-20 00:00:00.000000000 Z
+date: 2020-08-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -94,6 +94,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: dotenv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: foreman
   requirement: !ruby/object:Gem::Requirement
@@ -150,6 +164,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: puma
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
@@ -236,12 +264,14 @@ files:
 - README.md
 - exe/good_job
 - lib/active_job/queue_adapters/good_job_adapter.rb
+- lib/generators/good_job/install_generator.rb
+- lib/generators/good_job/templates/migration.rb.erb
 - lib/good_job.rb
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
-- lib/good_job/logging.rb
+- lib/good_job/log_subscriber.rb
 - lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb
@@ -256,7 +286,7 @@ metadata:
   documentation_uri: https://rdoc.info/github/bensheldon/good_job
   homepage_uri: https://github.com/bensheldon/good_job
   source_code_uri: https://github.com/bensheldon/good_job
-post_install_message: 
+post_install_message:
 rdoc_options:
 - "--title"
 - GoodJob - a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
@@ -279,7 +309,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.0.3
-signing_key: 
+signing_key:
 specification_version: 4
 summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
 test_files: []

data/lib/good_job/logging.rb

@@ -1,70 +0,0 @@
-module GoodJob
-  module Logging
-    extend ActiveSupport::Concern
-
-    included do
-      cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
-
-      def self.tag_logger(*tags)
-        if logger.respond_to?(:tagged)
-          tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
-          logger.tagged(*tags) { yield }
-        else
-          yield
-        end
-      end
-    end
-
-    class LogSubscriber < ActiveSupport::LogSubscriber
-      def create(event)
-        good_job = event.payload[:good_job]
-
-        info do
-          "Created GoodJob resource with id #{good_job.id}"
-        end
-      end
-
-      def timer_task_finished(event)
-        exception = event.payload[:error]
-        return unless exception
-
-        error do
-          "ERROR: #{exception}\n #{exception.backtrace}"
-        end
-      end
-
-      def job_finished(event)
-        exception = event.payload[:error]
-        return unless exception
-
-        error do
-          "ERROR: #{exception}\n #{exception.backtrace}"
-        end
-      end
-
-      def scheduler_start_shutdown(_event)
-        info do
-          "Shutting down scheduler..."
-        end
-      end
-
-      def scheduler_shutdown(_event)
-        info do
-          "Scheduler is shut down."
-        end
-      end
-
-      private
-
-      def logger
-        GoodJob.logger
-      end
-
-      def thread_name
-        Thread.current.name || Thread.current.object_id
-      end
-    end
-  end
-end
-
-GoodJob::Logging::LogSubscriber.attach_to :good_job