good_job 2.11.1 → 2.11.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c72b452a727d8f42ffb1a261669eb55a684ae5b9643d0f5660db7585d69416af
-  data.tar.gz: fe50c11e3b2c5c5aca4f86dadbfc8ab8c8bf3bd3f8898fccd25e4fb8ca47a151
+  metadata.gz: 44ffd48ff0a352758f67aeaaf60eeb7242fdf95c71cf16fcb7089689d72709a6
+  data.tar.gz: da29ba7f5b8f212d989d24fec8c7e5672166d3950bae04f3f29a2ca6e8cb035a
 SHA512:
-  metadata.gz: ead055bb27d8a25f3c199d731d6ad56390ac934a85dae640fb4aa2d35baae8aec0b0cb0e30354b2972591b69269caf30574df60d8ecbd3dca445170dd0e1598a
-  data.tar.gz: 396f2a76423df8033811186c993fad03007e25a1aaa85b3565b375d1a89b9934bcc4c42ebb7eda1fb1f5b00d89452a4315e642e155df4328a13fc25ec044f60e
+  metadata.gz: c72947a7302d52b2b4f81a6092fbaca28dc900095fa477e0f7abd6d3419c5825c9693063fcdf6081d00ed93b082be705cef2beb2d1088c90de54737098103df5
+  data.tar.gz: 4d9b6c34665cd02c840841460a2a162592d494fb27bd79e12987d4cda50a93d8959f130f29ccbad7e04b4be79ef5878dd13ae98ff87c6d8c87aad514b8937cd9

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [v2.11.2](https://github.com/bensheldon/good_job/tree/v2.11.2) (2022-03-03)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.1...v2.11.2)
+
+**Closed issues:**
+
+- Best practices in deploying and monitoring a queue [\#523](https://github.com/bensheldon/good_job/issues/523)
+
+**Merged pull requests:**
+
+- Wrap Rspec before and example blocks with a mutex for JRuby [\#537](https://github.com/bensheldon/good_job/pull/537) ([bensheldon](https://github.com/bensheldon))
+- Delegate `ActiveJobJob.table_name` to `Execution` and prevent it from being directly assignable [\#536](https://github.com/bensheldon/good_job/pull/536) ([bensheldon](https://github.com/bensheldon))
+- Enable DB table names customization [\#535](https://github.com/bensheldon/good_job/pull/535) ([dimvic](https://github.com/dimvic))
+- Added a chapter on how to prepare for production. [\#525](https://github.com/bensheldon/good_job/pull/525) ([stas](https://github.com/stas))
+
 ## [v2.11.1](https://github.com/bensheldon/good_job/tree/v2.11.1) (2022-03-01)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.0...v2.11.1)

data/README.md

@@ -52,6 +52,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Timeouts](#timeouts)
     - [Optimize queues, threads, and processes](#optimize-queues-threads-and-processes)
     - [Database connections](#database-connections)
+        - [Production setup](#production-setup)
     - [Execute jobs async / in-process](#execute-jobs-async--in-process)
     - [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
     - [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
@@ -674,13 +675,38 @@ Keep in mind, queue operations and management is an advanced discipline. This st
 
 ### Database connections
 
-Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. _Allowing GoodJob to create more threads than available database connections can lead to timeouts and is not recommended._ For example:
+Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. For example:
 
 ```yaml
 # config/database.yml
-pool: <%= [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i].max %>
+pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + (ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
 ```
 
+To calculate the total number of the database connections you'll need:
+
+- 1 connection dedicated to the scheduler aka `LISTEN/NOTIFY`
+- 1 connection per query pool thread e.g. `--queues=mice:2;elephants:1` is 3 threads. Pool thread size defaults to `--max-threads`
+- (optional) 2 connections for Cron scheduler if you're running it
+- (optional) 1 connection per subthread, if your application makes multithreaded database queries within a job
+- When running `:async`, you must also add the number of threads by the webserver
+
+The queue process will not crash if the connections pool is exhausted, instead it will report an exception (eg. `ActiveRecord::ConnectionTimeoutError`).
+
+#### Production setup
+
+When running GoodJob in a production environment, you should be mindful of:
+
+- [Execution mode](execute-jobs-async--in-process)
+- [Database connection pool size](#database-connections)
+- [Health check probes](#cli-http-health-check-probes) and potentially the [instrumentation support](#monitor-and-preserve-worked-jobs)
+
+The recommended way to monitor the queue in production is:
+
+- have an exception notifier callback (see `on_thread_error`)
+- if possible, run the queue as a dedicated instance and use available HTTP health check probes instead of pid-based monitoring
+- keep an eye on the number of jobs in the queue (abnormal high number of unscheduled jobs means the queue could be underperforming)
+- consider performance monitoring services which support the built-in Rails instrumentation (eg. Sentry, Skylight, etc.)
+
 ### Execute jobs async / in-process
 
 GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:

data/engine/app/charts/good_job/scheduled_by_queue_chart.rb

@@ -9,6 +9,7 @@ module GoodJob
     def data
       end_time = Time.current
       start_time = end_time - 1.day
+      table_name = GoodJob::ActiveJobJob.table_name
 
       count_query = Arel.sql(GoodJob::Execution.pg_or_jdbc_query(<<~SQL.squish))
         SELECT *
@@ -23,7 +24,7 @@ module GoodJob
               queue_name,
               count(*) AS count
             FROM (
-              #{@filter.filtered_query.except(:select, :order).select('queue_name', 'COALESCE(good_jobs.scheduled_at, good_jobs.created_at)::timestamp AS scheduled_at').to_sql}
+              #{@filter.filtered_query.except(:select, :order).select('queue_name', "COALESCE(#{table_name}.scheduled_at, #{table_name}.created_at)::timestamp AS scheduled_at").to_sql}
             ) sources
             GROUP BY date_trunc('hour', scheduled_at), queue_name
         ) sources ON sources.scheduled_at = timestamp

data/engine/app/filters/good_job/jobs_filter.rb

@@ -14,7 +14,7 @@ module GoodJob
 
     def filtered_query
       query = base_query.includes(:executions)
-                        .joins_advisory_locks.select('good_jobs.*', 'pg_locks.locktype AS locktype')
+                        .joins_advisory_locks.select("#{GoodJob::ActiveJobJob.table_name}.*", 'pg_locks.locktype AS locktype')
 
       query = query.job_class(params[:job_class]) if params[:job_class].present?
       query = query.where(queue_name: params[:queue_name]) if params[:queue_name].present?
@@ -31,7 +31,7 @@ module GoodJob
         when 'scheduled'
           query = query.scheduled
         when 'running'
-          query = query.running.select('good_jobs.*', 'pg_locks.locktype')
+          query = query.running.select("#{GoodJob::ActiveJobJob.table_name}.*", 'pg_locks.locktype')
         when 'queued'
           query = query.queued
         end

data/lib/good_job/active_job_job.rb

@@ -15,7 +15,14 @@ module GoodJob
     # Attached to a Job's Execution when the Job is discarded.
     DiscardJobError = Class.new(StandardError)
 
-    self.table_name = 'good_jobs'
+    class << self
+      delegate :table_name, to: Execution
+
+      def table_name=(_value)
+        raise NotImplementedError, 'Assign GoodJob::Execution.table_name directly'
+      end
+    end
+
     self.primary_key = 'active_job_id'
     self.advisory_lockable_column = 'active_job_id'
 

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '2.11.1'
+  VERSION = '2.11.2'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 2.11.1
+  version: 2.11.2
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-03-01 00:00:00.000000000 Z
+date: 2022-03-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob