good_job 1.0.3 → 1.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fc7b3997aaa38583efb4c91e66a2a5f9b8bfd75d4c14c9baaf8f1af7c236a42e
-  data.tar.gz: 285b441a5e5add440142960884370a1b74168266252fa48ce569cbb56e80f240
+  metadata.gz: 7134108a565e8b1efaa825cec90630cab0965fcfca5ff02f09751730f8839bdc
+  data.tar.gz: d887e3b6cbb6d3d877793b186de2c0ca5a65178b968b2000821c6f9000c0fdd1
 SHA512:
-  metadata.gz: f54e83d6ac19d80da2123a395ccf6ffd7eb2faab8e68bd5d50a572c8f0bf64fbc907125a47b7ff91dccac5497af113d043bba4f62ee6ae10fecaf3c004a1114d
-  data.tar.gz: 0b65a60bde26db0cfcfa55f166e80043bcbccb6e2c80f5de1840d2deae3ee28880c904bb53ffc2478ad834ffef8cfe567cee9a179932e20e4f92fdf8504838e4
+  metadata.gz: cce4a02b62ded08da0b2665cb346bfbeda6fdffa15182c8ab680962f6ba1f3534a8b9ac4cad8ecfa60d34b15cdfaabff5c8c80e4b3f7bdaa99c2d7a0e4208fe9
+  data.tar.gz: d9f13feb39a8700fe889c7b4250004095cdddeb5e1e9f06426610d10c13ce7c14d20db32810aa93dd4d1b7a075f585821bf4581634373a3585540bab0018f46c

data/CHANGELOG.md

@@ -1,6 +1,77 @@
 # Changelog
 
-## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-25)
+## [v1.1.4](https://github.com/bensheldon/good_job/tree/v1.1.4) (2020-08-19)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.3...v1.1.4)
+
+**Implemented enhancements:**
+
+- Explicitly name threads for easier debugging [\#64](https://github.com/bensheldon/good_job/issues/64)
+- Investigate Listen/Notify as alternative to polling [\#54](https://github.com/bensheldon/good_job/issues/54)
+
+**Merged pull requests:**
+
+- Add Postgres LISTEN/NOTIFY support [\#82](https://github.com/bensheldon/good_job/pull/82) ([bensheldon](https://github.com/bensheldon))
+- Allow Schedulers to filter \#create\_thread to avoid flood of queries when running async with multiple schedulers [\#81](https://github.com/bensheldon/good_job/pull/81) ([bensheldon](https://github.com/bensheldon))
+- Fully name scheduler threadpools and thread names; refactor CLI STDOUT [\#80](https://github.com/bensheldon/good_job/pull/80) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.3](https://github.com/bensheldon/good_job/tree/v1.1.3) (2020-08-14)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.2...v1.1.3)
+
+**Fixed bugs:**
+
+- Job exceptions not properly attached to good\_jobs record  [\#72](https://github.com/bensheldon/good_job/issues/72)
+
+**Merged pull requests:**
+
+- Capture errors via instrumentation from retry\_on and discard\_on [\#79](https://github.com/bensheldon/good_job/pull/79) ([bensheldon](https://github.com/bensheldon))
+- Document GoodJob::Scheduler with Yard [\#78](https://github.com/bensheldon/good_job/pull/78) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.2](https://github.com/bensheldon/good_job/tree/v1.1.2) (2020-08-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.1...v1.1.2)
+
+**Implemented enhancements:**
+
+- Allow the omission of queue names within a scheduler [\#73](https://github.com/bensheldon/good_job/issues/73)
+
+**Merged pull requests:**
+
+- Allow named queues to be excluded with a minus [\#77](https://github.com/bensheldon/good_job/pull/77) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.1](https://github.com/bensheldon/good_job/tree/v1.1.1) (2020-08-12)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.0...v1.1.1)
+
+**Implemented enhancements:**
+
+- Allow multiple schedulers within the same process. e.g. `queues=mice:2,elephants:4` [\#45](https://github.com/bensheldon/good_job/issues/45)
+
+**Merged pull requests:**
+
+- Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-10)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.3...v1.1.0)
+
+**Closed issues:**
+
+- Document reliability guarantees [\#59](https://github.com/bensheldon/good_job/issues/59)
+- Document how to hook in exception monitor \(Sentry, Rollbar, etc\) [\#47](https://github.com/bensheldon/good_job/issues/47)
+- Allow an Async mode [\#27](https://github.com/bensheldon/good_job/issues/27)
+
+**Merged pull requests:**
+
+- Add a callable hook on thread errors [\#71](https://github.com/bensheldon/good_job/pull/71) ([bensheldon](https://github.com/bensheldon))
+- Clarify reliability guarantees [\#70](https://github.com/bensheldon/good_job/pull/70) ([bensheldon](https://github.com/bensheldon))
+- Clean up Readme formatting; re-arrange tests for clarity and values [\#69](https://github.com/bensheldon/good_job/pull/69) ([bensheldon](https://github.com/bensheldon))
+- Create an Async execution mode [\#68](https://github.com/bensheldon/good_job/pull/68) ([bensheldon](https://github.com/bensheldon))
+- Move all stdout to LogSubscriber [\#67](https://github.com/bensheldon/good_job/pull/67) ([bensheldon](https://github.com/bensheldon))
+- Allow schedulers to be restarted; separate unit tests from integration tests [\#66](https://github.com/bensheldon/good_job/pull/66) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.3](https://github.com/bensheldon/good_job/tree/v1.0.3) (2020-07-26)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.2...v1.0.3)
 
@@ -14,6 +85,7 @@
 
 **Merged pull requests:**
 
+- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
 - Re-perform a job if a StandardError bubbles up; better document job reliability [\#62](https://github.com/bensheldon/good_job/pull/62) ([bensheldon](https://github.com/bensheldon))
 - Update the setup documentation to use correct bin setup command [\#61](https://github.com/bensheldon/good_job/pull/61) ([jm96441n](https://github.com/jm96441n))
 
@@ -25,10 +97,6 @@
 
 - Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
 
-**Closed issues:**
-
-- repeating/recurring jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
-
 **Merged pull requests:**
 
 - Add migration generator [\#56](https://github.com/bensheldon/good_job/pull/56) ([thedanbob](https://github.com/thedanbob))
@@ -40,7 +108,7 @@
 
 **Merged pull requests:**
 
-- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
 
 ## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
 
@@ -50,10 +118,6 @@
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.2...v0.9.0)
 
-**Merged pull requests:**
-
-- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
-
 ## [v0.8.2](https://github.com/bensheldon/good_job/tree/v0.8.2) (2020-07-18)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.1...v0.8.2)
@@ -81,6 +145,7 @@
 
 **Merged pull requests:**
 
+- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
 - Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-16)
@@ -113,7 +178,6 @@
 - Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
 - Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
 - Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
-- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
 
@@ -138,6 +202,7 @@
 
 **Merged pull requests:**
 
+- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
 - Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
 - Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
 - Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))

data/README.md

@@ -6,7 +6,7 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 
 - **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration. 
 - **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby). 
-- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
+- **Backed by Postgres.** Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`, and LISTEN/NOTIFY to reduce queuing latency.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
 For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
@@ -27,7 +27,8 @@ $ bundle install
 ## Usage
 
 1. Create a database migration:
-    ```bash
+    
+   ```bash
     $ bin/rails g good_job:install
     ```
 
@@ -38,7 +39,8 @@ $ bundle install
     ```
     
 1. Configure the ActiveJob adapter:
-    ```ruby
+    
+   ```ruby
     # config/application.rb
     config.active_job.queue_adapter = :good_job
     ```
@@ -57,31 +59,79 @@ $ bundle install
     ```
 
 1. Queue your job 🎉: 
+    
     ```ruby
     YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
+    
     ```bash
     $ bundle exec good_job
     ```
    
    Configuration options available with `help`:
-   ```bash
-   $ bundle exec good_job help start
+
+    ```bash
+    $ bundle exec good_job help start
    
-   # Usage:
-   #  good_job start
-   #
-   # Options:
-   #   [--max-threads=N]         # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
-   #   [--queues=queue1,queue2]  # Queues to work from. Separate multiple queues with commas (default: *)
-   #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
-   ```
+    Usage:
+      good_job start
+    
+    Options:
+      [--max-threads=N]                                          # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
+      [--queues=queue1,queue2(;queue3,queue4:5;-queue1,queue2)]  # Queues to work from. Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons (default: *)
+      [--poll-interval=N]                                        # Interval between polls for available jobs in seconds (default: 1)    
+    
+    Start job worker
+    ```
+
+1. Optimize execution to reduce congestion and execution latency. 
+
+    By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
+
+    - Multiple execution pools within a single process:
+        
+        ```bash
+        $ bundle exec good_job --queues=transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;* --max-threads=5
+        ```
+      
+        This configuration will result in a single process with 4 isolated thread execution pools. Isolated execution pools are separated with a semicolon (`;`) and queue names and thread counts with a colon (`:`)
+        
+        - `transactional_messages:2`: execute jobs enqueued on `transactional_messages` with up to 2 threads.
+        - `batch_processing:1` execute jobs enqueued on `batch_processing` with a single thread.
+        - `-transactional_messages,batch_processing`: execute jobs enqueued on _any_ queue _excluding_ `transactional_messages` or `batch_processing` with up to 2 threads.
+        - `*`: execute jobs on any queue on up to 5 threads, as configured by `--max-threads=5`
+        
+        For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy. 
+        
+        Configuration can be injected by environment variables too:
+        
+        ```bash
+        $ GOOD_JOB_QUEUES="transactional_messages:2;batch_processing:1;-transactional_messages,batch_processing:2;*" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
+        ```
+        
+    - Multiple processes; for example, on Heroku:
+    
+        ```procfile
+        # Procfile
+        
+        # Separate dyno types
+        worker: bundle exec good_job --max-threads=5
+        transactional_worker: bundle exec good_job --queues=transactional_messages --max-threads=2
+        batch_worker: bundle exec good_job --queues=batch_processing --max-threads=1
+      
+        # Combined multi-process dyno
+        combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues=transactional_messages --max-threads=2 & bundle exec good_job --queues=batch_processing --max-threads=1 & wait -n
+        ```
+      
+      Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
+
+    _Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job 👍_
 
 ### Error handling, retries, and reliability
 
-GoodJob guarantees _at-least-once_ performance of jobs. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. 
+GoodJob guarantees that a completely-performed job will run once and only once. GoodJob fully supports ActiveJob's built-in functionality for error handling, retries and timeouts. Writing reliable, transactional, and idempotent `ActiveJob#perform` methods is outside the scope of GoodJob.
 
 #### Error handling
 
@@ -95,6 +145,15 @@ By default, if a job raises an error while it is being performed, _and it bubble
     GoodJob.reperform_jobs_on_standard_error = true # => default
     ```
 
+To report errors that _do_ bubble up to the GoodJob backend, assign a callable to `GoodJob.on_thread_error`. For example:
+
+```ruby
+# config/initializers/good_job.rb
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+GoodJob.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
+```
+
 ### Retrying jobs
 
 ActiveJob can be configured to retry an infinite number of times, with an exponential backoff. Using ActiveJob's `retry_on` will ensure that errors do not bubble up to the GoodJob backend:
@@ -123,12 +182,29 @@ end
 
 GoodJob can be configured to allow omitting `retry_on`'s block argument and implicitly discard un-handled errors:
 
-    ```ruby
-    # config/initializers/good_job.rb
-    
-    # Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
-    GoodJob.reperform_jobs_on_standard_error = false 
-    ```
+```ruby
+# config/initializers/good_job.rb
+
+# Do NOT re-perform a job if a StandardError bubbles up to the GoodJob backend
+GoodJob.reperform_jobs_on_standard_error = false 
+```
+
+When using an exception monitoring service (e.g. Sentry, Bugsnag, Airbrake, Honeybadger, etc), the use of `rescue_on` may be incompatible with their ActiveJob integration. It's safest to explicitly wrap jobs with an exception reporter. For example:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+  
+  around_perform do |_job, block|
+    block.call
+  rescue StandardError => e
+    Raven.capture_exception(e)
+    raise
+  end
+  # ...
+end
+```
+
 
 ActiveJob's `discard_on` functionality is supported too.
 
@@ -139,6 +215,14 @@ Using a Mailer's `#deliver_later` will enqueue an instance of `ActionMailer::Del
 ```ruby
 # config/initializers/good_job.rb
 ActionMailer::DeliveryJob.retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+# With Sentry (or Bugsnag, Airbrake, Honeybadger, etc.)
+ActionMailer::DeliveryJob.around_perform do |_job, block|
+  block.call
+rescue StandardError => e
+  Raven.capture_exception(e)
+  raise
+end
 ```
 
 #### Timeouts
@@ -158,7 +242,7 @@ class ApplicationJob < ActiveJob::Base
 end
 ```
 
-### Configuring Job Execution Threads
+### Configuring job execution threads
     
 GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
 
@@ -169,6 +253,56 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Executing jobs async / in-process
+
+GoodJob is able to run "async" in the same process as the webserver (e.g. `bin/rail s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
+
+- Directly configure the ActiveJob adapter:
+
+    ```ruby
+    # config/environments/production.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :async, max_threads: 4, poll_interval: 30)
+    ```
+- Or, when using `...queue_adapter = :good_job`, via environment variables:
+
+    ```bash
+    $ GOOD_JOB_EXECUTION_MODE=async GOOD_JOB_MAX_THREADS=4 GOOD_JOB_POLL_INTERVAL=30 bin/rails server
+    ```
+ 
+Depending on your application configuration, you may need to take additional steps:
+
+- Ensure that you have enough database connections for both web and job execution threads:
+
+    ```yaml
+    # config/database.yml
+    pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
+    ```
+
+- When running Puma with workers (`WEB_CONCURRENCY > 0`) or another process-forking webserver, GoodJob's threadpool schedulers should be stopped before forking, restarted after fork, and cleanly shut down on exit. Stopping GoodJob's scheduler pre-fork is recommended to ensure that GoodJob does not continue executing jobs in the parent/controller process. For example, with Puma:
+
+    ```ruby
+    # config/puma.rb
+  
+    before_fork do
+      GoodJob.shutdown
+    end
+    
+    on_worker_boot do
+      GoodJob.restart
+    end
+    
+    on_worker_shutdown do
+      GoodJob.shutdown
+    end
+  
+    MAIN_PID = Process.pid
+    at_exit do
+      GoodJob.shutdown if Process.pid == MAIN_PID
+    end
+    ```
+  
+  GoodJob is compatible with Puma's `preload_app!` method.
+  
 ### Migrating to GoodJob from a different ActiveJob backend
 
 If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
@@ -184,7 +318,8 @@ If your application is already using an ActiveJob backend, you will need to inst
     ```
 
 1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
-    ```procfile
+    
+   ```procfile
     # Procfile
     # ...
     worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
@@ -210,15 +345,24 @@ It is also necessary to delete these preserved jobs from the database after a ce
 - For example, in a Rake task:
   
     ```ruby
-    # GoodJob::Job.finished(1.day.ago).delete_all
+    GoodJob::Job.finished(1.day.ago).delete_all
     ```
+
 - For example, using the `good_job` command-line utility:
 
     ```bash
     $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
     ```
 
-## Development
+## Contributing
+
+Contributions are welcomed and appreciated 🙏
+
+- Review the [Prioritized Project Backlog](https://github.com/bensheldon/good_job/projects/1).
+- Open a new Issue or contribute to an [existing Issue](https://github.com/bensheldon/good_job/issues). Questions or suggestions are fantastic.
+- Participate according to our [Code of Conduct](https://github.com/bensheldon/good_job/projects/1).
+
+### Gem development
 
 To run tests:
 
@@ -241,7 +385,6 @@ $ bundle exec appraisal
 
 # Run tests
 $ bundle exec appraisal bin/rspec
-
 ```
 
 For developing locally within another Ruby on Rails project:
@@ -256,24 +399,23 @@ $ bundle install
 # => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
 ```
 
-## Releasing
+### Releasing
 
-Package maintainers can release this gem with the following [gem-release](https://github.com/svenfuchs/gem-release) command:
+Package maintainers can release this gem by running:
 
 ```bash
 # Sign into rubygems
 $ gem signin
 
+# Add a .env file with the following:
+# CHANGELOG_GITHUB_TOKEN= # Github Personal Access Token
+
 # Update version number, changelog, and create git commit:
-$ bundle exec rake commit_version[minor] # major,minor,patch
+$ bundle exec rake release[minor] # major,minor,patch
 
 # ..and follow subsequent directions. 
 ```
 
-## Contributing
-
-Contribution directions go here.
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).