good_job 2.11.0 → 2.11.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4722bcae4ba953937f1e78dd730c61898ec0aa25e4c8d5fedc6163e218de4381
-  data.tar.gz: fe659df13800eadf8f8d45a2d31b4b4dd466d0f9517addefdfd5169c351c7c76
+  metadata.gz: 1c804e28bcaae78b4af5a654da6cacd9fba979159fb57cbbcfb4bcb348b947fa
+  data.tar.gz: cbd920c5050bcf5da2e328a3aa5c3bb638b5a077a3f67e6dbbf76202a38ec723
 SHA512:
-  metadata.gz: 3fff2b35292aebd982e6a11fac22d9a16b2468a19bf4e50c354a3f6806831fbcb2ab07d5cbe74c4ccd6bbeeb178abeeee700da0dbfdb0281f79b173b22a2ce9a
-  data.tar.gz: ea88184a6b3b3be56604b90af426c4c491f735dc5951854710bfeda55a1c4b930b76190845cb90b512d4e9e2f9a1f3b746478f232d10529c8a6971e448ab1419
+  metadata.gz: cef26c910377e43dc66facdb2ccbde95ace6b38c8a1a6672100496919573f3ce07f0e59cd68ee22f195f72590313ff66757086123be3beee07f0801ecf258bf8
+  data.tar.gz: ee5553f45df7ea0bb4eb7abb870a879556292a7b25247db8d40700347f639b98bfdc758105588a8059a07ac4d7a5d7ccc6e0ba5543daf5cae4e035431f021174

data/CHANGELOG.md

@@ -1,23 +1,74 @@
 # Changelog
 
+## [v2.11.3](https://github.com/bensheldon/good_job/tree/v2.11.3) (2022-03-30)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.2...v2.11.3)
+
+**Fixed bugs:**
+
+- Add explicit `kwargs:` key to cron configuration [\#548](https://github.com/bensheldon/good_job/pull/548) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- ArgumentError: wrong number of arguments \(given 1, expected 0; required keyword: schedule\) - cron [\#546](https://github.com/bensheldon/good_job/issues/546)
+- How to run clean up preserved jobs in cron? [\#541](https://github.com/bensheldon/good_job/issues/541)
+- Erroring with "Too many open files" when good\_job tries reconnecting to database [\#530](https://github.com/bensheldon/good_job/issues/530)
+- Can't cast Array [\#529](https://github.com/bensheldon/good_job/issues/529)
+
+**Merged pull requests:**
+
+- Use bundle add instead [\#542](https://github.com/bensheldon/good_job/pull/542) ([glaucocustodio](https://github.com/glaucocustodio))
+- Update Readme to better explain queues, pools, threads, and database connections; update CLI to frontload queue option [\#539](https://github.com/bensheldon/good_job/pull/539) ([bensheldon](https://github.com/bensheldon))
+
+## [v2.11.2](https://github.com/bensheldon/good_job/tree/v2.11.2) (2022-03-03)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.1...v2.11.2)
+
+**Closed issues:**
+
+- Best practices in deploying and monitoring a queue [\#523](https://github.com/bensheldon/good_job/issues/523)
+
+**Merged pull requests:**
+
+- Wrap Rspec before and example blocks with a mutex for JRuby [\#537](https://github.com/bensheldon/good_job/pull/537) ([bensheldon](https://github.com/bensheldon))
+- Delegate `ActiveJobJob.table_name` to `Execution` and prevent it from being directly assignable [\#536](https://github.com/bensheldon/good_job/pull/536) ([bensheldon](https://github.com/bensheldon))
+- Enable DB table names customization [\#535](https://github.com/bensheldon/good_job/pull/535) ([dimvic](https://github.com/dimvic))
+- Added a chapter on how to prepare for production. [\#525](https://github.com/bensheldon/good_job/pull/525) ([stas](https://github.com/stas))
+
+## [v2.11.1](https://github.com/bensheldon/good_job/tree/v2.11.1) (2022-03-01)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.0...v2.11.1)
+
+**Fixed bugs:**
+
+- Ensure sticky footer doesn't overlap paginater; fix polling interval to 30 seconds, not ms [\#534](https://github.com/bensheldon/good_job/pull/534) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Pagination buttons hidden behind footer [\#533](https://github.com/bensheldon/good_job/issues/533)
+
 ## [v2.11.0](https://github.com/bensheldon/good_job/tree/v2.11.0) (2022-02-27)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.10.0...v2.11.0)
 
+**Implemented enhancements:**
+
+- Add support for live polling the dashboard [\#528](https://github.com/bensheldon/good_job/pull/528) ([danielwestendorf](https://github.com/danielwestendorf))
+
 **Closed issues:**
 
 - How do I ensure that a the same job can't run twice? \(unique job / avoid duplicates\) [\#531](https://github.com/bensheldon/good_job/issues/531)
 - Bulk reschedule and discard jobs via dashboard [\#527](https://github.com/bensheldon/good_job/issues/527)
 - "Live Poll" dashboard [\#526](https://github.com/bensheldon/good_job/issues/526)
 
-**Merged pull requests:**
-
-- Add support for live polling the dashboard [\#528](https://github.com/bensheldon/good_job/pull/528) ([danielwestendorf](https://github.com/danielwestendorf))
-
 ## [v2.10.0](https://github.com/bensheldon/good_job/tree/v2.10.0) (2022-02-18)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.9.6...v2.10.0)
 
+**Implemented enhancements:**
+
+- Dashboard: update search filters and some small UI updates [\#518](https://github.com/bensheldon/good_job/pull/518) ([multiplegeorges](https://github.com/multiplegeorges))
+
 **Closed issues:**
 
 - Cron jobs not getting run [\#519](https://github.com/bensheldon/good_job/issues/519)
@@ -29,7 +80,6 @@
 - Fix Benchmark job throughput script   [\#522](https://github.com/bensheldon/good_job/pull/522) ([douglara](https://github.com/douglara))
 - Update development Gemfile.lock [\#521](https://github.com/bensheldon/good_job/pull/521) ([bensheldon](https://github.com/bensheldon))
 - Ensure Rails 6.0 is tested against Ruby 3.0; use Ruby 3.0 in demo environment [\#520](https://github.com/bensheldon/good_job/pull/520) ([bensheldon](https://github.com/bensheldon))
-- Dashboard: update search filters and some small UI updates [\#518](https://github.com/bensheldon/good_job/pull/518) ([multiplegeorges](https://github.com/multiplegeorges))
 - Document safer setting for retry\_on\_unhandled\_error [\#517](https://github.com/bensheldon/good_job/pull/517) ([tamaloa](https://github.com/tamaloa))
 
 ## [v2.9.6](https://github.com/bensheldon/good_job/tree/v2.9.6) (2022-02-07)

data/README.md

@@ -52,6 +52,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Timeouts](#timeouts)
     - [Optimize queues, threads, and processes](#optimize-queues-threads-and-processes)
     - [Database connections](#database-connections)
+        - [Production setup](#production-setup)
     - [Execute jobs async / in-process](#execute-jobs-async--in-process)
     - [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
     - [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
@@ -64,16 +65,10 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
 
 ## Set up
 
-1. Add `good_job` to your application's Gemfile:
+1. Add `good_job` to your application's Gemfile and install the gem:
 
-    ```ruby
-    gem 'good_job'
-    ```
-
-1. Install the gem:
-
-    ```bash
-    bundle install
+    ```sh
+    bundle add good_job
     ```
 
 1. Run the GoodJob install generator. This will generate a database migration to create a table for GoodJob's job records:
@@ -164,8 +159,8 @@ Usage:
   good_job start
 
 Options:
-  [--max-threads=COUNT]        # Maximum number of threads to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)
-  [--queues=QUEUE_LIST]        # Queues to work from. (env var: GOOD_JOB_QUEUES, default: *)
+  [--queues=QUEUE_LIST]        # Queues or pools to work from. (env var: GOOD_JOB_QUEUES, default: *)
+  [--max-threads=COUNT]        # Default number of threads per pool to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)
   [--poll-interval=SECONDS]    # Interval between polls for available jobs in seconds (env var: GOOD_JOB_POLL_INTERVAL, default: 1)
   [--max-cache=COUNT]          # Maximum number of scheduled jobs to cache in memory (env var: GOOD_JOB_MAX_CACHE, default: 10000)
   [--shutdown-timeout=SECONDS] # Number of seconds to wait for jobs to finish when shutting down before stopping the thread. (env var: GOOD_JOB_SHUTDOWN_TIMEOUT, default: -1 (forever))
@@ -229,12 +224,12 @@ Rails.application.configure do
   config.good_job.retry_on_unhandled_error = false
   config.good_job.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
   config.good_job.execution_mode = :async
+  config.good_job.queues = '*'
   config.good_job.max_threads = 5
   config.good_job.poll_interval = 30 # seconds
   config.good_job.shutdown_timeout = 25 # seconds
   config.good_job.enable_cron = true
   config.good_job.cron = { example: { cron: '0 * * * *', class: 'ExampleJob'  } }
-  config.good_job.queues = '*'
 
   # ...or all at once.
   config.good_job = {
@@ -242,6 +237,7 @@ Rails.application.configure do
     retry_on_unhandled_error: false,
     on_thread_error: -> (exception) { Raven.capture_exception(exception) },
     execution_mode: :async,
+    queues: '*',
     max_threads: 5,
     poll_interval: 30,
     shutdown_timeout: 25,
@@ -252,7 +248,6 @@ Rails.application.configure do
         class: 'ExampleJob'
       },
     },
-    queues: '*',
   }
 end
 ```
@@ -264,8 +259,8 @@ Available configuration options are:
     - `:external` causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you’ll need to use the command-line tool to actually execute your jobs.
     - `:async` (or `:async_server`) executes jobs in separate threads within the Rails web server process (`bundle exec rails server`). It can be more economical for small workloads because you don’t need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose `:external` instead.  When not in the Rails web server, jobs will execute in `:external` mode to ensure jobs are not executed within `rails console`, `rails db:migrate`, `rails assets:prepare`, etc.
     - `:async_all` executes jobs in separate threads in _any_ Rails process.
-- `max_threads` (integer) sets the maximum number of threads to use when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
-- `queues` (string) determines which queues to execute jobs from when `execution_mode` is set to `:async`. See the description of `good_job start` for more details on the format of this string. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
+- `queues` (string) sets queues or pools to execute jobs. You can also set this with the environment variable `GOOD_JOB_QUEUES`.
+- `max_threads` (integer) sets the default number of threads per pool to use for working jobs. You can also set this with the environment variable `GOOD_JOB_MAX_THREADS`.
 - `poll_interval` (integer) sets the number of seconds between polls for jobs when `execution_mode` is set to `:async`. You can also set this with the environment variable `GOOD_JOB_POLL_INTERVAL`. A poll interval of `-1` disables polling completely.
 - `max_cache` (integer) sets the maximum number of scheduled jobs that will be stored in memory to reduce execution latency when also polling for scheduled jobs. Caching 10,000 scheduled jobs uses approximately 20MB of memory. You can also set this with the environment variable `GOOD_JOB_MAX_CACHE`.
 - `shutdown_timeout` (float) number of seconds to wait for jobs to finish when shutting down before stopping the thread. Defaults to forever: `-1`. You can also set this with the environment variable `GOOD_JOB_SHUTDOWN_TIMEOUT`.
@@ -441,7 +436,8 @@ config.good_job.cron = {
   frequent_task: { # each recurring job must have a unique key
     cron: "*/15 * * * *", # cron-style scheduling format by fugit gem
     class: "ExampleJob", # reference the Job class with a string
-    args: [42, { name: "Alice" }], # arguments to pass; can also be a proc e.g. `-> { { when: Time.now } }`
+    args: [42, "life"], # positional arguments to pass; can also be a proc e.g. `-> { [Time.now] }`
+    kwargs: { name: "Alice" }, # keyword arguments to pass; can also be a proc e.g. `-> { { name: NAMES.sample } }`
     set: { priority: -10 }, # additional ActiveJob properties; can also be a lambda/proc e.g. `-> { { priority: [1,2].sample } }`
     description: "Something helpful", # optional description that appears in Dashboard (coming soon!)
   },
@@ -674,13 +670,38 @@ Keep in mind, queue operations and management is an advanced discipline. This st
 
 ### Database connections
 
-Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. _Allowing GoodJob to create more threads than available database connections can lead to timeouts and is not recommended._ For example:
+Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. For example:
 
 ```yaml
 # config/database.yml
-pool: <%= [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i].max %>
+pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + 3 + (ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i %>
 ```
 
+To calculate the total number of the database connections you'll need:
+
+- 1 connection dedicated to the scheduler aka `LISTEN/NOTIFY`
+- 1 connection per query pool thread e.g. `--queues=mice:2;elephants:1` is 3 threads. Pool thread size defaults to `--max-threads`
+- (optional) 2 connections for Cron scheduler if you're running it
+- (optional) 1 connection per subthread, if your application makes multithreaded database queries within a job
+- When running `:async`, you must also add the number of threads by the webserver
+
+The queue process will not crash if the connections pool is exhausted, instead it will report an exception (eg. `ActiveRecord::ConnectionTimeoutError`).
+
+#### Production setup
+
+When running GoodJob in a production environment, you should be mindful of:
+
+- [Execution mode](execute-jobs-async--in-process)
+- [Database connection pool size](#database-connections)
+- [Health check probes](#cli-http-health-check-probes) and potentially the [instrumentation support](#monitor-and-preserve-worked-jobs)
+
+The recommended way to monitor the queue in production is:
+
+- have an exception notifier callback (see `on_thread_error`)
+- if possible, run the queue as a dedicated instance and use available HTTP health check probes instead of pid-based monitoring
+- keep an eye on the number of jobs in the queue (abnormal high number of unscheduled jobs means the queue could be underperforming)
+- consider performance monitoring services which support the built-in Rails instrumentation (eg. Sentry, Skylight, etc.)
+
 ### Execute jobs async / in-process
 
 GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:

data/engine/app/assets/scripts.js

@@ -37,13 +37,13 @@ const GoodJob = {
     // live poll interval and enablement
     if (urlParams.has('poll')) {
       const parsedInterval = (parseInt(urlParams.get('poll')) || GOOD_JOB_DEFAULT_POLL_INTERVAL_SECONDS) * 1000;
-
-      GoodJob.pollEnabled = true;
       GoodJob.pollInterval = Math.max(parsedInterval, GOOD_JOB_MINIMUM_POLL_INTERVAL);
       GoodJob.setStorage('pollInterval', GoodJob.pollInterval);
+
+      GoodJob.pollEnabled = true;
     } else {
+      GoodJob.pollInterval = GoodJob.getStorage('pollInterval') || (GOOD_JOB_DEFAULT_POLL_INTERVAL_SECONDS * 1000);
       GoodJob.pollEnabled = GoodJob.getStorage('pollEnabled') || false;
-      GoodJob.pollInterval = GoodJob.getStorage('pollInterval') || GOOD_JOB_DEFAULT_POLL_INTERVAL_SECONDS;
     }
 
     document.getElementById('toggle-poll').checked = GoodJob.pollEnabled;

data/engine/app/assets/style.css

@@ -23,3 +23,8 @@
   position: relative;
   height: 200px;
 }
+
+body {
+  /* Make room for the sticky footer */
+  margin-bottom: 100px;
+}

data/engine/app/charts/good_job/scheduled_by_queue_chart.rb

@@ -9,6 +9,7 @@ module GoodJob
     def data
       end_time = Time.current
       start_time = end_time - 1.day
+      table_name = GoodJob::ActiveJobJob.table_name
 
       count_query = Arel.sql(GoodJob::Execution.pg_or_jdbc_query(<<~SQL.squish))
         SELECT *
@@ -23,7 +24,7 @@ module GoodJob
               queue_name,
               count(*) AS count
             FROM (
-              #{@filter.filtered_query.except(:select, :order).select('queue_name', 'COALESCE(good_jobs.scheduled_at, good_jobs.created_at)::timestamp AS scheduled_at').to_sql}
+              #{@filter.filtered_query.except(:select, :order).select('queue_name', "COALESCE(#{table_name}.scheduled_at, #{table_name}.created_at)::timestamp AS scheduled_at").to_sql}
             ) sources
             GROUP BY date_trunc('hour', scheduled_at), queue_name
         ) sources ON sources.scheduled_at = timestamp

data/engine/app/filters/good_job/jobs_filter.rb

@@ -14,7 +14,7 @@ module GoodJob
 
     def filtered_query
       query = base_query.includes(:executions)
-                        .joins_advisory_locks.select('good_jobs.*', 'pg_locks.locktype AS locktype')
+                        .joins_advisory_locks.select("#{GoodJob::ActiveJobJob.table_name}.*", 'pg_locks.locktype AS locktype')
 
       query = query.job_class(params[:job_class]) if params[:job_class].present?
       query = query.where(queue_name: params[:queue_name]) if params[:queue_name].present?
@@ -31,7 +31,7 @@ module GoodJob
         when 'scheduled'
           query = query.scheduled
         when 'running'
-          query = query.running.select('good_jobs.*', 'pg_locks.locktype')
+          query = query.running.select("#{GoodJob::ActiveJobJob.table_name}.*", 'pg_locks.locktype')
         when 'queued'
           query = query.queued
         end

data/lib/good_job/active_job_job.rb

@@ -15,7 +15,14 @@ module GoodJob
     # Attached to a Job's Execution when the Job is discarded.
     DiscardJobError = Class.new(StandardError)
 
-    self.table_name = 'good_jobs'
+    class << self
+      delegate :table_name, to: Execution
+
+      def table_name=(_value)
+        raise NotImplementedError, 'Assign GoodJob::Execution.table_name directly'
+      end
+    end
+
     self.primary_key = 'active_job_id'
     self.advisory_lockable_column = 'active_job_id'
 

data/lib/good_job/cli.rb

@@ -50,14 +50,14 @@ module GoodJob
       separate isolated execution pools with semicolons and threads with colons.
 
     DESCRIPTION
-    method_option :max_threads,
-                  type: :numeric,
-                  banner: 'COUNT',
-                  desc: "Maximum number of threads to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)"
     method_option :queues,
                   type: :string,
                   banner: "QUEUE_LIST",
-                  desc: "Queues to work from. (env var: GOOD_JOB_QUEUES, default: *)"
+                  desc: "Queues or queue pools to work from. (env var: GOOD_JOB_QUEUES, default: *)"
+    method_option :max_threads,
+                  type: :numeric,
+                  banner: 'COUNT',
+                  desc: "Default number of threads per pool to use for working jobs. (env var: GOOD_JOB_MAX_THREADS, default: 5)"
     method_option :poll_interval,
                   type: :numeric,
                   banner: 'SECONDS',

data/lib/good_job/cron_entry.rb

@@ -52,6 +52,10 @@ module GoodJob # :nodoc:
       params[:args]
     end
 
+    def kwargs
+      params[:kwargs]
+    end
+
     def description
       params[:description]
     end
@@ -87,7 +91,8 @@ module GoodJob # :nodoc:
         current_thread.cron_key = key
         current_thread.cron_at = cron_at
 
-        job_class.constantize.set(set_value).perform_later(*args_value)
+        configured_job = job_class.constantize.set(set_value)
+        kwargs_value.present? ? configured_job.perform_later(*args_value, **kwargs_value) : configured_job.perform_later(*args_value)
       end
     rescue ActiveRecord::RecordNotUnique
       false
@@ -124,6 +129,11 @@ module GoodJob # :nodoc:
       value.respond_to?(:call) ? value.call : value
     end
 
+    def kwargs_value
+      value = kwargs || nil
+      value.respond_to?(:call) ? value.call : value
+    end
+
     def display_property(value)
       case value
       when NilClass

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '2.11.0'
+  VERSION = '2.11.3'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 2.11.0
+  version: 2.11.3
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-02-28 00:00:00.000000000 Z
+date: 2022-03-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob