good_job 1.0.1 → 1.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9e068306c80c1080f736520219cdf27230f9684bd69fa70cff0489ce37b1ed79
-  data.tar.gz: 8f10fb768248ca1ec1e14f44be2073f9c60069b2451c5858c571efbca84c0969
+  metadata.gz: 4117e28d2a899ebb8b66a17fa79ff8495b36521d92f9146059c7b78406a361b6
+  data.tar.gz: 535dd8b9204cc76ee6749224217e5d33e23adc2414e8ca7c420d4deb35451efd
 SHA512:
-  metadata.gz: 8ebe33709d71b2254fa2fc0dbc1c991df739a67ba6ee3649000bada6c4618b2d958c58372818502b57211fb4694b28b16be513227220b262da2125b6f87b61bb
-  data.tar.gz: 40db6accac1448cef4cfff7516e68d401b25711520d479cf3af4c20f03a31526f2e084aed5dabc64adceb9b23ae75ac41ae74e25bb80a4bb80c320a31fc9fb25
+  metadata.gz: 1f8bf7bf0f21604eb08b814d032b4a964f220b6d6384b368de7263672ece002f8fd01e5e2d77b0c6ed93e31ac03f235a2c40b8ad3f532de3fb3c755cd629ad24
+  data.tar.gz: 6006611999250e6b7708f381302379f1a07ba3522151630b708388436c9264ea2e2b4277c5753027171e82e7dd8a82b6f272386ad39170a192b94d0b36e45111

data/CHANGELOG.md

@@ -1,6 +1,80 @@
 # Changelog
 
-## [v1.0.1](https://github.com/bensheldon/good_job/tree/v1.0.1) (2020-07-21)
+## [v1.1.2](https://github.com/bensheldon/good_job/tree/v1.1.2) (2020-08-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.1...v1.1.2)
+
+**Implemented enhancements:**
+
+- Allow the omission of queue names within a scheduler [\#73](https://github.com/bensheldon/good_job/issues/73)
+
+**Merged pull requests:**
+
+- Allow named queues to be excluded with a minus [\#77](https://github.com/bensheldon/good_job/pull/77) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.1](https://github.com/bensheldon/good_job/tree/v1.1.1) (2020-08-12)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.0...v1.1.1)
+
+**Implemented enhancements:**
+
+- Allow multiple schedulers within the same process. e.g. `queues=mice:2,elephants:4` [\#45](https://github.com/bensheldon/good_job/issues/45)
+
+**Merged pull requests:**
+
+- Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
+- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-10)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.3...v1.1.0)
+
+**Closed issues:**
+
+- Document reliability guarantees [\#59](https://github.com/bensheldon/good_job/issues/59)
+- Document how to hook in exception monitor \(Sentry, Rollbar, etc\) [\#47](https://github.com/bensheldon/good_job/issues/47)
+- Allow an Async mode [\#27](https://github.com/bensheldon/good_job/issues/27)
+
+**Merged pull requests:**
+
+- Add a callable hook on thread errors [\#71](https://github.com/bensheldon/good_job/pull/71) ([bensheldon](https://github.com/bensheldon))
+- Clarify reliability guarantees [\#70](https://github.com/bensheldon/good_job/pull/70) ([bensheldon](https://github.com/bensheldon))
+- Clean up Readme formatting; re-arrange tests for clarity and values [\#69](https://github.com/bensheldon/good_job/pull/69) ([bensheldon](https://github.com/bensheldon))
+- Create an Async execution mode [\#68](https://github.com/bensheldon/good_job/pull/68) ([bensheldon](https://github.com/bensheldon))
+- Move all stdout to LogSubscriber [\#67](https://github.com/bensheldon/good_job/pull/67) ([bensheldon](https://github.com/bensheldon))
+- Allow schedulers to be restarted; separate unit tests from integration tests [\#66](https://github.com/bensheldon/good_job/pull/66) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-26)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.2...v1.0.3)
+
+**Fixed bugs:**
+
+- Preserve GoodJob::Jobs when a StandardError is raised [\#60](https://github.com/bensheldon/good_job/issues/60)
+
+**Closed issues:**
+
+- Have an initial setup generator [\#6](https://github.com/bensheldon/good_job/issues/6)
+
+**Merged pull requests:**
+
+- Re-perform a job if a StandardError bubbles up; better document job reliability [\#62](https://github.com/bensheldon/good_job/pull/62) ([bensheldon](https://github.com/bensheldon))
+- Update the setup documentation to use correct bin setup command [\#61](https://github.com/bensheldon/good_job/pull/61) ([jm96441n](https://github.com/jm96441n))
+
+## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-25)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.1...v1.0.2)
+
+**Fixed bugs:**
+
+- Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
+
+**Merged pull requests:**
+
+- Add migration generator [\#56](https://github.com/bensheldon/good_job/pull/56) ([thedanbob](https://github.com/thedanbob))
+- Fix migration script in readme [\#55](https://github.com/bensheldon/good_job/pull/55) ([thedanbob](https://github.com/thedanbob))
+
+## [v1.0.1](https://github.com/bensheldon/good_job/tree/v1.0.1) (2020-07-22)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.0...v1.0.1)
 
@@ -32,12 +106,15 @@
 
 - Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
 - Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
-- Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.8.1](https://github.com/bensheldon/good_job/tree/v0.8.1) (2020-07-18)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.0...v0.8.1)
 
+**Merged pull requests:**
+
+- Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
+
 ## [v0.8.0](https://github.com/bensheldon/good_job/tree/v0.8.0) (2020-07-17)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.7.0...v0.8.0)

data/README.md

@@ -9,6 +9,8 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 - **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
+For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -25,35 +27,11 @@ $ bundle install
 ## Usage
 
 1. Create a database migration:
-    ```bash
-    $ bin/rails g migration CreateGoodJobs
+    
+   ```bash
+    $ bin/rails g good_job:install
     ```
 
-    Add to the newly created migration file:
-
-    ```ruby
-    class CreateGoodJobs < ActiveRecord::Migration[6.0]
-      def change
-        enable_extension 'pgcrypto'
-
-        create_table :good_jobs, id: :uuid do |t|
-          t.timestamps
-
-          t.text :queue_name
-          t.integer :priority
-          t.jsonb :serialized_params
-          t.timestamp :scheduled_at
-          t.timestamp :performed_at
-          t.timestamp :finished_at
-          t.text :error
-
-          add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
-          add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
-        end
-      end
-    end
-    ```
-    
     Run the migration:
     
     ```bash
@@ -61,7 +39,8 @@ $ bundle install
     ```
     
 1. Configure the ActiveJob adapter:
-    ```ruby
+    
+   ```ruby
     # config/application.rb
     config.active_job.queue_adapter = :good_job
     ```
@@ -80,48 +59,190 @@ $ bundle install
     ```
 
 1. Queue your job 🎉: 
+    
     ```ruby
     YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
+    
     ```bash
     $ bundle exec good_job
     ```
    
    Configuration options available with `help`:
-   ```bash
-   $ bundle exec good_job help start
+
+    ```bash
+    $ bundle exec good_job help start
+   
+    Usage:
+      good_job start
+    
+    Options:
+      [--max-threads=N]                                          # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
+      [--queues=queue1,queue2(;queue3,queue4:5;-queue1,queue2)]  # Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)
+      [--poll-interval=N]                                        # Interval between polls for available jobs in seconds (default: 1)    
+    
+    Start job worker
+    ```
+
+1. Optimize execution to reduce congestion and execution latency. 
+
+    By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
+
+    - Multiple execution pools within a single process:
+        
+        ```bash
+        $ bundle exec good_job --queues=transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;* --max-threads=5
+        ```
+      
+        This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
+        
+        - `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
+        - `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
+        - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
+        - `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
+        
+        For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy. 
+        
+        Configuration can be injected by environment variables too:
+        
+        ```bash
+        $ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
+        ```
+        
+    - Multiple processes; for example, on Heroku:
+    
+        ```procfile
+        # Procfile
+        
+        # Separate dyno types
+        worker: bundle exec good_job --max-threads=5
+        transactional_worker: bundle exec good_job --queues=transactional_messages --max-threads=2
+        batch_worker: bundle exec good_job --queues=batch_processing --max-threads=1
+      
+        # Combined multi-process dyno
+        combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues=transactional_messages --max-threads=2 & bundle exec good_job --queues=batch_processing --max-threads=1 & wait -n
+        ```
+      
+      Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
+
+    _Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job 👍_
+
+### Error handling, retries, and reliability
+
+GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. Writing reliable, transactional, and idempotent `ActiveJob#perform` methods is outside the scope of GoodJob.
+
+#### Error handling
+
+By default, if a job raises an error while it is being performed, _and it bubbles up to the GoodJob backend_, GoodJob will be immediately re-perform the job until it finishes successfully.
+
+- `Exception`-type errors, such as a SIGINT, will always cause a job to be re-performed.
+- `StandardError`-type errors, by default, will cause a job to be re-performed, though this is configurable:
    
-   # Usage:
-   #  good_job start
-   #
-   # Options:
-   #   [--max-threads=N]         # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
-   #   [--queues=queue1,queue2]  # Queues to work from. Separate multiple queues with commas (default: *)
-   #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
-   ```
+    ```ruby
+    # config/initializers/good_job.rb
+    GoodJob.reperform_jobs_on_standard_error = true # => default
+    ```
+
+To report errors that _do_ bubble up to the GoodJob backend, assign a callable to `GoodJob.on_thread_error`. For example:
+
+```ruby
+# config/initializers/good_job.rb
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+```
+
+### Retrying jobs
+
+ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  # ...
+end
+```
+
+When specifying a limited number of retries, care must be taken to ensure that an error does not bubble up to the GoodJob backend because that will result in the job being re-performed:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, attempts: 5 do |_job, _exception|
+    # Log error, etc.
+    # You must implement this block, otherwise, 
+    #   Active Job will re-raise the error.
+    # Do not re-raise the error, otherwise 
+    #   GoodJob will immediately re-perform the job. 
+  end
+  # ...
+end
+```
 
-### Taking advantage of ActiveJob
+GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
+
+```ruby
+# config/initializers/good_job.rb
+
+# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
+GoodJob.reperform_jobs_on_standard_error = false 
+```
 
-ActiveJob has a rich set of built-in functionality for timeouts, error handling, and retrying. For example:
+When using an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), the use of `rescue_on` may be incompatible with their ActiveJob integration. It's safest to explicitly wrap jobs with an exception reporter. For example:
 
 ```ruby
 class ApplicationJob < ActiveJob::Base  
-  # Retry errors an infinite number of times with exponential back-off
   retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  
+  around_perform do |_job, block|
+    block.call
+  rescue StandardError => e
+    Raven.capture_exception(e)
+    raise
+  end
+  # ...
+end
+```
+
+
+ActiveJob's `discard_on` functionality is supported too.
+
+#### ActionMailer retries
+
+Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::DeliveryJob` which inherits from `ActiveJob::Base` rather than your applications `ApplicationJob`. You can use an initializer to configure retries on `ActionMailer::DeliveryJob`:
+
+```ruby
+# config/initializers/good_job.rb
+ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+ActionMailer::DeliveryJob.around_perform do |_job, block|
+  block.call
+rescue StandardError => e
+  Raven.capture_exception(e)
+  raise
+end
+```
 
-  # Timeout jobs after 10 minutes
+#### Timeouts
+
+Job timeouts can be configured with an `around_perform`:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
   JobTimeoutError = Class.new(StandardError)
+  
   around_perform do |_job, block|
+    # Timeout jobs after 10 minutes
     Timeout.timeout(10.minutes, JobTimeoutError) do
       block.call
     end
   end
 end
 ```
-   
-### Configuring Job Execution Threads
+
+### Configuring job execution threads
     
 GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
 
@@ -132,6 +253,51 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Executing jobs async / in-process
+
+GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
+
+- Directly configure the ActiveJob adapter:
+
+    ```ruby
+    # config/environments/production.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
+    ```
+- Or, when using `...queue_adapter = :good_job`, via environment variables:
+
+    ```bash
+    $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
+    ```
+ 
+Depending on your application configuration, you may need to take additional steps:
+
+- Ensure that you have enough database connections for both web and job execution threads:
+
+    ```yaml
+    # config/database.yml
+    pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
+    ```
+
+- When running Puma with workers (`WEB_CONCURRENCY > 0`) or another process-forking webserver, GoodJob's threadpool schedulers should be stopped before forking, restarted after fork, and cleanly shut down on exit. Stopping GoodJob's scheduler pre-fork is recommended to ensure that GoodJob does not continue executing jobs in the parent/controller process. For example, with Puma:
+
+    ```ruby
+    # config/puma.rb
+  
+    before_fork do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    
+    on_worker_boot do
+      GoodJob::Scheduler.instances.each { |s| s.restart }
+    end
+    
+    on_worker_shutdown do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    ```
+  
+  GoodJob is compatible with Puma's `preload_app!` method.
+  
 ### Migrating to GoodJob from a different ActiveJob backend
 
 If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
@@ -147,7 +313,8 @@ If your application is already using an ActiveJob backend, you will need to inst
     ```
 
 1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
-    ```procfile
+    
+   ```procfile
     # Procfile
     # ...
     worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
@@ -173,15 +340,24 @@ It is also necessary to delete these preserved jobs from the database after a ce
 - For example, in a Rake task:
   
     ```ruby
-    # GoodJob::Job.finished(1.day.ago).delete_all
+    GoodJob::Job.finished(1.day.ago).delete_all
     ```
+
 - For example, using the `good_job` command-line utility:
 
     ```bash
     $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
     ```
 
-## Development
+## Contributing
+
+Contributions are welcomed and appreciated 🙏
+
+- Review the [Prioritized Project Backlog](https://github.com/bensheldon/good_job/projects/1).
+- Open a new Issue or contribute to an [existing Issue](https://github.com/bensheldon/good_job/issues). Questions or suggestions are fantastic.
+- Participate according to our [Code of Conduct](https://github.com/bensheldon/good_job/projects/1).
+
+### Gem development
 
 To run tests:
 
@@ -190,7 +366,7 @@ To run tests:
 $ git clone git@github.com:bensheldon/good_job.git
 
 # Set up the local environment
-$ bin/setup_test
+$ bin/setup
 
 # Run the tests
 $ bin/rspec
@@ -204,7 +380,6 @@ $ bundle exec appraisal
 
 # Run tests
 $ bundle exec appraisal bin/rspec
-
 ```
 
 For developing locally within another Ruby on Rails project:
@@ -219,24 +394,23 @@ $ bundle install
 # => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
 ```
 
-## Releasing
+### Releasing
 
-Package maintainers can release this gem with the following [gem-release](https://github.com/svenfuchs/gem-release) command:
+Package maintainers can release this gem by running:
 
 ```bash
 # Sign into rubygems
 $ gem signin
 
+# Add a .env file with the following:
+# CHANGELOG_GITHUB_TOKEN= # Github Personal Access Token
+
 # Update version number, changelog, and create git commit:
-$ bundle exec rake commit_version[minor] # major,minor,patch
+$ bundle exec rake release[minor] # major,minor,patch
 
 # ..and follow subsequent directions. 
 ```
 
-## Contributing
-
-Contribution directions go here.
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).