good_job 1.0.2 → 1.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f7c64e8766a52e6bbbb5e55b409bb0100e54790d6d88340893e61f930cbf8b9e
-  data.tar.gz: 7e89b7bf2d3aaa45cc5b07deab36f5334a61cac44c0aec06206ddbd6eb0ebc22
+  metadata.gz: 44da5dd6eb7dee7e08319f7405e97f81e4223f8c88d48ba4a0512fa006f89e78
+  data.tar.gz: fe3f5ca8a39b85b1aa4be77a819e910691223cf36e80c6ab337ff1b224a26e16
 SHA512:
-  metadata.gz: 944b64f713ad584b56fe386dc63aa6afc0538e498cff4fb46aec9098306b7e436a3d29cebd968157f34e62c9b1b0ce932e15967b4a631f9f04c60ca43d77f4c8
-  data.tar.gz: 5000ffc97bc381cb94999478c1fe90202b681d5cf87d697f4700cc60b94a7417fd67ca3f5428c4b50acb195bde7850a3a5d809e32c5baaf08ae3a58a2101624a
+  metadata.gz: e6648e41c7ff99915716702651cd43fb7fffcff24528a9fb2d7e4a7913303d85d25c3b3f8ef54cc1b716a273836bc3c7fb2911750e1d99f216b2074b03f13ae8
+  data.tar.gz: '063048f09b76b10d4f38beb5fcf390d972f0952ed4df7b1053e8078da2bad005b816e21271026b95f8eaef12b8e3e1ce0a4cecf2cf40f965e852a8e2e9854aac'

data/CHANGELOG.md

@@ -1,16 +1,87 @@
 # Changelog
 
-## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-24)
+## [v1.1.3](https://github.com/bensheldon/good_job/tree/v1.1.3) (2020-08-14)
 
-[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.1...v1.0.2)
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.2...v1.1.3)
 
 **Fixed bugs:**
 
-- Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
+- Job exceptions not properly attached to good\_jobs record  [\#72](https://github.com/bensheldon/good_job/issues/72)
+
+**Merged pull requests:**
+
+- Capture errors via instrumentation from retry\_on and discard\_on [\#79](https://github.com/bensheldon/good_job/pull/79) ([bensheldon](https://github.com/bensheldon))
+- Document GoodJob::Scheduler with Yard [\#78](https://github.com/bensheldon/good_job/pull/78) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.2](https://github.com/bensheldon/good_job/tree/v1.1.2) (2020-08-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.1...v1.1.2)
+
+**Implemented enhancements:**
+
+- Allow the omission of queue names within a scheduler [\#73](https://github.com/bensheldon/good_job/issues/73)
+
+**Merged pull requests:**
+
+- Allow named queues to be excluded with a minus [\#77](https://github.com/bensheldon/good_job/pull/77) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.1](https://github.com/bensheldon/good_job/tree/v1.1.1) (2020-08-12)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.0...v1.1.1)
+
+**Implemented enhancements:**
+
+- Allow multiple schedulers within the same process. e.g. `queues=mice:2,elephants:4` [\#45](https://github.com/bensheldon/good_job/issues/45)
+
+**Merged pull requests:**
+
+- Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
+- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-10)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.3...v1.1.0)
 
 **Closed issues:**
 
-- repeating/recurring jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
+- Document reliability guarantees [\#59](https://github.com/bensheldon/good_job/issues/59)
+- Document how to hook in exception monitor \(Sentry, Rollbar, etc\) [\#47](https://github.com/bensheldon/good_job/issues/47)
+- Allow an Async mode [\#27](https://github.com/bensheldon/good_job/issues/27)
+
+**Merged pull requests:**
+
+- Add a callable hook on thread errors [\#71](https://github.com/bensheldon/good_job/pull/71) ([bensheldon](https://github.com/bensheldon))
+- Clarify reliability guarantees [\#70](https://github.com/bensheldon/good_job/pull/70) ([bensheldon](https://github.com/bensheldon))
+- Clean up Readme formatting; re-arrange tests for clarity and values [\#69](https://github.com/bensheldon/good_job/pull/69) ([bensheldon](https://github.com/bensheldon))
+- Create an Async execution mode [\#68](https://github.com/bensheldon/good_job/pull/68) ([bensheldon](https://github.com/bensheldon))
+- Move all stdout to LogSubscriber [\#67](https://github.com/bensheldon/good_job/pull/67) ([bensheldon](https://github.com/bensheldon))
+- Allow schedulers to be restarted; separate unit tests from integration tests [\#66](https://github.com/bensheldon/good_job/pull/66) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-26)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.2...v1.0.3)
+
+**Fixed bugs:**
+
+- Preserve GoodJob::Jobs when a StandardError is raised [\#60](https://github.com/bensheldon/good_job/issues/60)
+
+**Closed issues:**
+
+- Have an initial setup generator [\#6](https://github.com/bensheldon/good_job/issues/6)
+
+**Merged pull requests:**
+
+- Re-perform a job if a StandardError bubbles up; better document job reliability [\#62](https://github.com/bensheldon/good_job/pull/62) ([bensheldon](https://github.com/bensheldon))
+- Update the setup documentation to use correct bin setup command [\#61](https://github.com/bensheldon/good_job/pull/61) ([jm96441n](https://github.com/jm96441n))
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-25)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.1...v1.0.2)
+
+**Fixed bugs:**
+
+- Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
 
 **Merged pull requests:**
 
@@ -33,10 +104,6 @@
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.2...v0.9.0)
 
-**Merged pull requests:**
-
-- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
-
 ## [v0.8.2](https://github.com/bensheldon/good_job/tree/v0.8.2) (2020-07-18)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.1...v0.8.2)

data/README.md

@@ -27,7 +27,8 @@ $ bundle install
 ## Usage
 
 1. Create a database migration:
-    ```bash
+    
+   ```bash
     $ bin/rails g good_job:install
     ```
 
@@ -38,7 +39,8 @@ $ bundle install
     ```
     
 1. Configure the ActiveJob adapter:
-    ```ruby
+    
+   ```ruby
     # config/application.rb
     config.active_job.queue_adapter = :good_job
     ```
@@ -57,48 +59,190 @@ $ bundle install
     ```
 
 1. Queue your job 🎉: 
+    
     ```ruby
     YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
+    
     ```bash
     $ bundle exec good_job
     ```
    
    Configuration options available with `help`:
-   ```bash
-   $ bundle exec good_job help start
+
+    ```bash
+    $ bundle exec good_job help start
    
-   # Usage:
-   #  good_job start
-   #
-   # Options:
-   #   [--max-threads=N]         # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
-   #   [--queues=queue1,queue2]  # Queues to work from. Separate multiple queues with commas (default: *)
-   #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
-   ```
+    Usage:
+      good_job start
+    
+    Options:
+      [--max-threads=N]                                          # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
+      [--queues=queue1,queue2(;queue3,queue4:5;-queue1,queue2)]  # Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)
+      [--poll-interval=N]                                        # Interval between polls for available jobs in seconds (default: 1)    
+    
+    Start job worker
+    ```
 
-### Taking advantage of ActiveJob
+1. Optimize execution to reduce congestion and execution latency. 
+
+    By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
+
+    - Multiple execution pools within a single process:
+        
+        ```bash
+        $ bundle exec good_job --queues=transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;* --max-threads=5
+        ```
+      
+        This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
+        
+        - `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
+        - `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
+        - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
+        - `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
+        
+        For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy. 
+        
+        Configuration can be injected by environment variables too:
+        
+        ```bash
+        $ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
+        ```
+        
+    - Multiple processes; for example, on Heroku:
+    
+        ```procfile
+        # Procfile
+        
+        # Separate dyno types
+        worker: bundle exec good_job --max-threads=5
+        transactional_worker: bundle exec good_job --queues=transactional_messages --max-threads=2
+        batch_worker: bundle exec good_job --queues=batch_processing --max-threads=1
+      
+        # Combined multi-process dyno
+        combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues=transactional_messages --max-threads=2 & bundle exec good_job --queues=batch_processing --max-threads=1 & wait -n
+        ```
+      
+      Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
 
-ActiveJob has a rich set of built-in functionality for timeouts, error handling, and retrying. For example:
+    _Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job 👍_
+
+### Error handling, retries, and reliability
+
+GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. Writing reliable, transactional, and idempotent `ActiveJob#perform` methods is outside the scope of GoodJob.
+
+#### Error handling
+
+By default, if a job raises an error while it is being performed, _and it bubbles up to the GoodJob backend_, GoodJob will be immediately re-perform the job until it finishes successfully.
+
+- `Exception`-type errors, such as a SIGINT, will always cause a job to be re-performed.
+- `StandardError`-type errors, by default, will cause a job to be re-performed, though this is configurable:
+   
+    ```ruby
+    # config/initializers/good_job.rb
+    GoodJob.reperform_jobs_on_standard_error = true # => default
+    ```
+
+To report errors that _do_ bubble up to the GoodJob backend, assign a callable to `GoodJob.on_thread_error`. For example:
+
+```ruby
+# config/initializers/good_job.rb
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+```
+
+### Retrying jobs
+
+ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  # ...
+end
+```
+
+When specifying a limited number of retries, care must be taken to ensure that an error does not bubble up to the GoodJob backend because that will result in the job being re-performed:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, attempts: 5 do |_job, _exception|
+    # Log error, etc.
+    # You must implement this block, otherwise, 
+    #   Active Job will re-raise the error.
+    # Do not re-raise the error, otherwise 
+    #   GoodJob will immediately re-perform the job. 
+  end
+  # ...
+end
+```
+
+GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
+
+```ruby
+# config/initializers/good_job.rb
+
+# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
+GoodJob.reperform_jobs_on_standard_error = false 
+```
+
+When using an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), the use of `rescue_on` may be incompatible with their ActiveJob integration. It's safest to explicitly wrap jobs with an exception reporter. For example:
 
 ```ruby
 class ApplicationJob < ActiveJob::Base  
-  # Retry errors an infinite number of times with exponential back-off
   retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  
+  around_perform do |_job, block|
+    block.call
+  rescue StandardError => e
+    Raven.capture_exception(e)
+    raise
+  end
+  # ...
+end
+```
+
+
+ActiveJob's `discard_on` functionality is supported too.
+
+#### ActionMailer retries
+
+Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::DeliveryJob` which inherits from `ActiveJob::Base` rather than your applications `ApplicationJob`. You can use an initializer to configure retries on `ActionMailer::DeliveryJob`:
+
+```ruby
+# config/initializers/good_job.rb
+ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+ActionMailer::DeliveryJob.around_perform do |_job, block|
+  block.call
+rescue StandardError => e
+  Raven.capture_exception(e)
+  raise
+end
+```
 
-  # Timeout jobs after 10 minutes
+#### Timeouts
+
+Job timeouts can be configured with an `around_perform`:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
   JobTimeoutError = Class.new(StandardError)
+  
   around_perform do |_job, block|
+    # Timeout jobs after 10 minutes
     Timeout.timeout(10.minutes, JobTimeoutError) do
       block.call
     end
   end
 end
 ```
-   
-### Configuring Job Execution Threads
+
+### Configuring job execution threads
     
 GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
 
@@ -109,6 +253,51 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Executing jobs async / in-process
+
+GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
+
+- Directly configure the ActiveJob adapter:
+
+    ```ruby
+    # config/environments/production.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
+    ```
+- Or, when using `...queue_adapter = :good_job`, via environment variables:
+
+    ```bash
+    $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
+    ```
+ 
+Depending on your application configuration, you may need to take additional steps:
+
+- Ensure that you have enough database connections for both web and job execution threads:
+
+    ```yaml
+    # config/database.yml
+    pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
+    ```
+
+- When running Puma with workers (`WEB_CONCURRENCY > 0`) or another process-forking webserver, GoodJob's threadpool schedulers should be stopped before forking, restarted after fork, and cleanly shut down on exit. Stopping GoodJob's scheduler pre-fork is recommended to ensure that GoodJob does not continue executing jobs in the parent/controller process. For example, with Puma:
+
+    ```ruby
+    # config/puma.rb
+  
+    before_fork do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    
+    on_worker_boot do
+      GoodJob::Scheduler.instances.each { |s| s.restart }
+    end
+    
+    on_worker_shutdown do
+      GoodJob::Scheduler.instances.each { |s| s.shutdown }
+    end
+    ```
+  
+  GoodJob is compatible with Puma's `preload_app!` method.
+  
 ### Migrating to GoodJob from a different ActiveJob backend
 
 If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
@@ -124,7 +313,8 @@ If your application is already using an ActiveJob backend, you will need to inst
     ```
 
 1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
-    ```procfile
+    
+   ```procfile
     # Procfile
     # ...
     worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
@@ -150,15 +340,24 @@ It is also necessary to delete these preserved jobs from the database after a ce
 - For example, in a Rake task:
   
     ```ruby
-    # GoodJob::Job.finished(1.day.ago).delete_all
+    GoodJob::Job.finished(1.day.ago).delete_all
     ```
+
 - For example, using the `good_job` command-line utility:
 
     ```bash
     $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
     ```
 
-## Development
+## Contributing
+
+Contributions are welcomed and appreciated 🙏
+
+- Review the [Prioritized Project Backlog](https://github.com/bensheldon/good_job/projects/1).
+- Open a new Issue or contribute to an [existing Issue](https://github.com/bensheldon/good_job/issues). Questions or suggestions are fantastic.
+- Participate according to our [Code of Conduct](https://github.com/bensheldon/good_job/projects/1).
+
+### Gem development
 
 To run tests:
 
@@ -167,7 +366,7 @@ To run tests:
 $ git clone git@github.com:bensheldon/good_job.git
 
 # Set up the local environment
-$ bin/setup_test
+$ bin/setup
 
 # Run the tests
 $ bin/rspec
@@ -181,7 +380,6 @@ $ bundle exec appraisal
 
 # Run tests
 $ bundle exec appraisal bin/rspec
-
 ```
 
 For developing locally within another Ruby on Rails project:
@@ -196,24 +394,23 @@ $ bundle install
 # => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
 ```
 
-## Releasing
+### Releasing
 
-Package maintainers can release this gem with the following [gem-release](https://github.com/svenfuchs/gem-release) command:
+Package maintainers can release this gem by running:
 
 ```bash
 # Sign into rubygems
 $ gem signin
 
+# Add a .env file with the following:
+# CHANGELOG_GITHUB_TOKEN= # Github Personal Access Token
+
 # Update version number, changelog, and create git commit:
-$ bundle exec rake commit_version[minor] # major,minor,patch
+$ bundle exec rake release[minor] # major,minor,patch
 
 # ..and follow subsequent directions. 
 ```
 
-## Contributing
-
-Contribution directions go here.
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).