good_job 2.10.0 → 2.11.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6b22cbc6d37af9dd8db290a1dc8073d2cae14c63e5cf7180b06dce1a33fb9992
-  data.tar.gz: 8cd87b2d69afea84db87125bad625558783adea98fee138be852e6a51c43dc50
+  metadata.gz: 44ffd48ff0a352758f67aeaaf60eeb7242fdf95c71cf16fcb7089689d72709a6
+  data.tar.gz: da29ba7f5b8f212d989d24fec8c7e5672166d3950bae04f3f29a2ca6e8cb035a
 SHA512:
-  metadata.gz: 230b335c5b6a03397c3212cd2bc59d87b2c735073233d068cf2c3045933b2aace533b0699a6fa1ecd41d77890af5aa18e1685b2c85f4c97c868773a3211736b7
-  data.tar.gz: b3fb4cb11f129fa8576e36c4e7401cdab927cf309bda963c3b82480dbe92663847b46a8e641b123e5ad7b28c53aa361d5fb6968131e23e85b0302768cc38e5d0
+  metadata.gz: c72947a7302d52b2b4f81a6092fbaca28dc900095fa477e0f7abd6d3419c5825c9693063fcdf6081d00ed93b082be705cef2beb2d1088c90de54737098103df5
+  data.tar.gz: 4d9b6c34665cd02c840841460a2a162592d494fb27bd79e12987d4cda50a93d8959f130f29ccbad7e04b4be79ef5878dd13ae98ff87c6d8c87aad514b8937cd9

data/CHANGELOG.md

@@ -1,9 +1,54 @@
 # Changelog
 
+## [v2.11.2](https://github.com/bensheldon/good_job/tree/v2.11.2) (2022-03-03)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.1...v2.11.2)
+
+**Closed issues:**
+
+- Best practices in deploying and monitoring a queue [\#523](https://github.com/bensheldon/good_job/issues/523)
+
+**Merged pull requests:**
+
+- Wrap Rspec before and example blocks with a mutex for JRuby [\#537](https://github.com/bensheldon/good_job/pull/537) ([bensheldon](https://github.com/bensheldon))
+- Delegate `ActiveJobJob.table_name` to `Execution` and prevent it from being directly assignable [\#536](https://github.com/bensheldon/good_job/pull/536) ([bensheldon](https://github.com/bensheldon))
+- Enable DB table names customization [\#535](https://github.com/bensheldon/good_job/pull/535) ([dimvic](https://github.com/dimvic))
+- Added a chapter on how to prepare for production. [\#525](https://github.com/bensheldon/good_job/pull/525) ([stas](https://github.com/stas))
+
+## [v2.11.1](https://github.com/bensheldon/good_job/tree/v2.11.1) (2022-03-01)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.11.0...v2.11.1)
+
+**Fixed bugs:**
+
+- Ensure sticky footer doesn't overlap paginater; fix polling interval to 30 seconds, not ms [\#534](https://github.com/bensheldon/good_job/pull/534) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- Pagination buttons hidden behind footer [\#533](https://github.com/bensheldon/good_job/issues/533)
+
+## [v2.11.0](https://github.com/bensheldon/good_job/tree/v2.11.0) (2022-02-27)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v2.10.0...v2.11.0)
+
+**Implemented enhancements:**
+
+- Add support for live polling the dashboard [\#528](https://github.com/bensheldon/good_job/pull/528) ([danielwestendorf](https://github.com/danielwestendorf))
+
+**Closed issues:**
+
+- How do I ensure that a the same job can't run twice? \(unique job / avoid duplicates\) [\#531](https://github.com/bensheldon/good_job/issues/531)
+- Bulk reschedule and discard jobs via dashboard [\#527](https://github.com/bensheldon/good_job/issues/527)
+- "Live Poll" dashboard [\#526](https://github.com/bensheldon/good_job/issues/526)
+
 ## [v2.10.0](https://github.com/bensheldon/good_job/tree/v2.10.0) (2022-02-18)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.9.6...v2.10.0)
 
+**Implemented enhancements:**
+
+- Dashboard: update search filters and some small UI updates [\#518](https://github.com/bensheldon/good_job/pull/518) ([multiplegeorges](https://github.com/multiplegeorges))
+
 **Closed issues:**
 
 - Cron jobs not getting run [\#519](https://github.com/bensheldon/good_job/issues/519)
@@ -15,7 +60,6 @@
 - Fix Benchmark job throughput script   [\#522](https://github.com/bensheldon/good_job/pull/522) ([douglara](https://github.com/douglara))
 - Update development Gemfile.lock [\#521](https://github.com/bensheldon/good_job/pull/521) ([bensheldon](https://github.com/bensheldon))
 - Ensure Rails 6.0 is tested against Ruby 3.0; use Ruby 3.0 in demo environment [\#520](https://github.com/bensheldon/good_job/pull/520) ([bensheldon](https://github.com/bensheldon))
-- Dashboard: update search filters and some small UI updates [\#518](https://github.com/bensheldon/good_job/pull/518) ([multiplegeorges](https://github.com/multiplegeorges))
 - Document safer setting for retry\_on\_unhandled\_error [\#517](https://github.com/bensheldon/good_job/pull/517) ([tamaloa](https://github.com/tamaloa))
 
 ## [v2.9.6](https://github.com/bensheldon/good_job/tree/v2.9.6) (2022-02-07)

data/README.md

@@ -52,6 +52,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Timeouts](#timeouts)
     - [Optimize queues, threads, and processes](#optimize-queues-threads-and-processes)
     - [Database connections](#database-connections)
+        - [Production setup](#production-setup)
     - [Execute jobs async / in-process](#execute-jobs-async--in-process)
     - [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
     - [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
@@ -369,6 +370,10 @@ GoodJob includes a Dashboard as a mountable `Rails::Engine`.
     end
     ```
 
+#### Live Polling
+
+The Dashboard can be set to automatically refresh by checking "Live Poll" in the Dashboard header, or by setting `?poll=10` with the interval in seconds (default 30 seconds).
+
 ### ActiveJob concurrency
 
 GoodJob can extend ActiveJob to provide limits on concurrently running jobs, either at time of _enqueue_ or at _perform_. Limiting concurrency can help prevent duplicate, double or unecessary jobs from being enqueued, or race conditions when performing, for example when interacting with 3rd-party APIs.
@@ -670,13 +675,38 @@ Keep in mind, queue operations and management is an advanced discipline. This st
 
 ### Database connections
 
-Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. _Allowing GoodJob to create more threads than available database connections can lead to timeouts and is not recommended._ For example:
+Each GoodJob execution thread requires its own database connection that is automatically checked out from Rails’ connection pool. For example:
 
 ```yaml
 # config/database.yml
-pool: <%= [ENV.fetch("RAILS_MAX_THREADS", 5).to_i, ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i].max %>
+pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + (ENV.fetch("GOOD_JOB_MAX_THREADS", 4).to_i %>
 ```
 
+To calculate the total number of the database connections you'll need:
+
+- 1 connection dedicated to the scheduler aka `LISTEN/NOTIFY`
+- 1 connection per query pool thread e.g. `--queues=mice:2;elephants:1` is 3 threads. Pool thread size defaults to `--max-threads`
+- (optional) 2 connections for Cron scheduler if you're running it
+- (optional) 1 connection per subthread, if your application makes multithreaded database queries within a job
+- When running `:async`, you must also add the number of threads by the webserver
+
+The queue process will not crash if the connections pool is exhausted, instead it will report an exception (eg. `ActiveRecord::ConnectionTimeoutError`).
+
+#### Production setup
+
+When running GoodJob in a production environment, you should be mindful of:
+
+- [Execution mode](execute-jobs-async--in-process)
+- [Database connection pool size](#database-connections)
+- [Health check probes](#cli-http-health-check-probes) and potentially the [instrumentation support](#monitor-and-preserve-worked-jobs)
+
+The recommended way to monitor the queue in production is:
+
+- have an exception notifier callback (see `on_thread_error`)
+- if possible, run the queue as a dedicated instance and use available HTTP health check probes instead of pid-based monitoring
+- keep an eye on the number of jobs in the queue (abnormal high number of unscheduled jobs means the queue could be underperforming)
+- consider performance monitoring services which support the built-in Rails instrumentation (eg. Sentry, Skylight, etc.)
+
 ### Execute jobs async / in-process
 
 GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:

data/engine/app/assets/scripts.js

@@ -1 +1,133 @@
-GoodJob = {};
+/*jshint esversion: 6, strict: false */
+const GOOD_JOB_DEFAULT_POLL_INTERVAL_SECONDS = 30;
+const GOOD_JOB_MINIMUM_POLL_INTERVAL = 1000;
+
+const GoodJob = {
+  // Register functions to execute when the DOM is ready
+  ready: (callback) => {
+    if (document.readyState !== "loading") {
+      callback();
+    } else {
+      document.addEventListener("DOMContentLoaded", callback);
+    }
+  },
+
+  init: () => {
+    GoodJob.updateSettings();
+    GoodJob.addListeners();
+    GoodJob.pollUpdates();
+    GoodJob.renderCharts(true);
+  },
+
+  addListeners: () => {
+    const gjActionEls = document.querySelectorAll('[data-gj-action]');
+
+    for (let i = 0; i < gjActionEls.length; i++) {
+      const el = gjActionEls[i];
+      const [eventName, func] = el.dataset.gjAction.split('#');
+
+      el.addEventListener(eventName, GoodJob[func]);
+    }
+  },
+
+  updateSettings: () => {
+    const queryString = window.location.search;
+    const urlParams = new URLSearchParams(queryString);
+
+    // live poll interval and enablement
+    if (urlParams.has('poll')) {
+      const parsedInterval = (parseInt(urlParams.get('poll')) || GOOD_JOB_DEFAULT_POLL_INTERVAL_SECONDS) * 1000;
+      GoodJob.pollInterval = Math.max(parsedInterval, GOOD_JOB_MINIMUM_POLL_INTERVAL);
+      GoodJob.setStorage('pollInterval', GoodJob.pollInterval);
+
+      GoodJob.pollEnabled = true;
+    } else {
+      GoodJob.pollInterval = GoodJob.getStorage('pollInterval') || (GOOD_JOB_DEFAULT_POLL_INTERVAL_SECONDS * 1000);
+      GoodJob.pollEnabled = GoodJob.getStorage('pollEnabled') || false;
+    }
+
+    document.getElementById('toggle-poll').checked = GoodJob.pollEnabled;
+  },
+
+  togglePoll: (ev) => {
+    GoodJob.pollEnabled = ev.currentTarget.checked;
+    GoodJob.setStorage('pollEnabled', GoodJob.pollEnabled);
+  },
+
+  pollUpdates: () => {
+    setTimeout(() => {
+      if (GoodJob.pollEnabled === true) {
+        fetch(window.location.href)
+          .then(resp => resp.text())
+          .then(GoodJob.updateContent)
+          .finally(GoodJob.pollUpdates);
+      } else {
+        GoodJob.pollUpdates();
+      }
+    }, GoodJob.pollInterval);
+  },
+
+  updateContent: (newContent) => {
+    const domParser = new DOMParser();
+    const parsedDOM = domParser.parseFromString(newContent, "text/html");
+
+    const newElements = parsedDOM.querySelectorAll('[data-gj-poll-replace]');
+
+    for (let i = 0; i < newElements.length; i++) {
+      const newEl = newElements[i];
+      const oldEl = document.getElementById(newEl.id);
+
+      if (oldEl) {
+        oldEl.replaceWith(newEl);
+      }
+    }
+
+    GoodJob.renderCharts(false);
+  },
+
+  renderCharts: (animate) => {
+    const charts = document.querySelectorAll('.chart');
+
+    for (let i = 0; i < charts.length; i++) {
+      const chartEl = charts[i];
+      const chartData = JSON.parse(chartEl.dataset.json);
+
+      const ctx = chartEl.getContext('2d');
+      const chart = new Chart(ctx, {
+        type: 'line',
+        data: {
+          labels: chartData.labels,
+          datasets: chartData.datasets
+        },
+        options: {
+          animation: animate,
+          responsive: true,
+          maintainAspectRatio: false,
+          scales: {
+            y: {
+              beginAtZero: true
+            }
+          }
+        }
+      });
+    }
+  },
+
+  getStorage: (key) => {
+    const value = localStorage.getItem('good_job-' + key);
+
+    if (value === 'true') {
+      return true;
+    } else if (value === 'false') {
+      return false;
+    } else {
+      return value;
+    }
+  },
+
+  setStorage: (key, value) => {
+    localStorage.setItem('good_job-' + key, value);
+  }
+};
+
+GoodJob.ready(GoodJob.init);

data/engine/app/assets/style.css

@@ -23,3 +23,8 @@
   position: relative;
   height: 200px;
 }
+
+body {
+  /* Make room for the sticky footer */
+  margin-bottom: 100px;
+}

data/engine/app/charts/good_job/scheduled_by_queue_chart.rb

@@ -9,6 +9,7 @@ module GoodJob
     def data
       end_time = Time.current
       start_time = end_time - 1.day
+      table_name = GoodJob::ActiveJobJob.table_name
 
       count_query = Arel.sql(GoodJob::Execution.pg_or_jdbc_query(<<~SQL.squish))
         SELECT *
@@ -23,7 +24,7 @@ module GoodJob
               queue_name,
               count(*) AS count
             FROM (
-              #{@filter.filtered_query.except(:select, :order).select('queue_name', 'COALESCE(good_jobs.scheduled_at, good_jobs.created_at)::timestamp AS scheduled_at').to_sql}
+              #{@filter.filtered_query.except(:select, :order).select('queue_name', "COALESCE(#{table_name}.scheduled_at, #{table_name}.created_at)::timestamp AS scheduled_at").to_sql}
             ) sources
             GROUP BY date_trunc('hour', scheduled_at), queue_name
         ) sources ON sources.scheduled_at = timestamp

data/engine/app/filters/good_job/jobs_filter.rb

@@ -14,7 +14,7 @@ module GoodJob
 
     def filtered_query
       query = base_query.includes(:executions)
-                        .joins_advisory_locks.select('good_jobs.*', 'pg_locks.locktype AS locktype')
+                        .joins_advisory_locks.select("#{GoodJob::ActiveJobJob.table_name}.*", 'pg_locks.locktype AS locktype')
 
       query = query.job_class(params[:job_class]) if params[:job_class].present?
       query = query.where(queue_name: params[:queue_name]) if params[:queue_name].present?
@@ -31,7 +31,7 @@ module GoodJob
         when 'scheduled'
           query = query.scheduled
         when 'running'
-          query = query.running.select('good_jobs.*', 'pg_locks.locktype')
+          query = query.running.select("#{GoodJob::ActiveJobJob.table_name}.*", 'pg_locks.locktype')
         when 'queued'
           query = query.queued
         end

data/engine/app/views/good_job/executions/_table.erb

@@ -1,4 +1,4 @@
-<div class="card my-3">
+<div class="card my-3" data-gj-poll-replace id="executions-table">
   <div class="table-responsive">
     <table class="table card-table table-bordered table-hover table-sm mb-0" id="executions_index_table">
       <thead>

data/engine/app/views/good_job/executions/index.html.erb

@@ -1,4 +1,4 @@
-<div class="card my-3 p-6">
+<div class="card my-3 p-6" data-gj-poll-replace id="executions-chart">
   <%= render 'good_job/shared/chart', chart_data: GoodJob::ScheduledByQueueChart.new(@filter).data %>
 </div>
 
@@ -7,7 +7,7 @@
 <%= render 'good_job/executions/table', executions: @filter.records %>
 
 <% if @filter.records.present? %>
-  <nav aria-label="Job pagination" class="mt-3">
+  <nav aria-label="Job pagination" class="mt-3" data-gj-poll-replace id="executions-pagination">
     <ul class="pagination">
       <li class="page-item">
         <%= link_to({ after_scheduled_at: (@filter.last.scheduled_at || @filter.last.created_at), after_id: @filter.last.id }, class: "page-link") do %>

data/engine/app/views/good_job/jobs/_table.erb

@@ -1,4 +1,4 @@
-<div class="card my-3">
+<div class="card my-3" data-gj-poll-replace id="jobs-table">
   <div class="table-responsive">
     <table class="table card-table table-bordered table-hover table-sm mb-0">
       <thead>

data/engine/app/views/good_job/jobs/index.html.erb

@@ -2,7 +2,7 @@
   <h2>All Jobs</h2>
 </div>
 
-<div class="card my-3 p-6">
+<div class="card my-3 p-6" data-gj-poll-replace id="jobs-chart">
   <%= render 'good_job/shared/chart', chart_data: GoodJob::ScheduledByQueueChart.new(@filter).data %>
 </div>
 
@@ -11,7 +11,7 @@
 <%= render 'good_job/jobs/table', jobs: @filter.records %>
 
 <% if @filter.records.present? %>
-  <nav aria-label="Job pagination" class="mt-3">
+  <nav aria-label="Job pagination" class="mt-3" data-gj-poll-replace id="jobs-pagination">
     <ul class="pagination">
       <li class="page-item">
         <%= link_to(@filter.to_params(after_scheduled_at: (@filter.last.scheduled_at || @filter.last.created_at), after_id: @filter.last.id), class: "page-link") do %>

data/engine/app/views/good_job/processes/index.html.erb

@@ -2,43 +2,45 @@
   <h2>Processes</h2>
 </div>
 
-<% if !GoodJob::Process.migrated? %>
-  <div class="card my-3">
-    <div class="card-body">
-      <p class="card-text">
-        <em>Feature unavailable because of pending database migration.</em>
-      </p>
+<div data-gj-poll-replace id="processes">
+  <% if !GoodJob::Process.migrated? %>
+    <div class="card my-3">
+      <div class="card-body">
+        <p class="card-text">
+          <em>Feature unavailable because of pending database migration.</em>
+        </p>
+      </div>
     </div>
-  </div>
-<% elsif @processes.present? %>
-  <div class="card my-3">
-    <div class="table-responsive">
-      <table class="table card-table table-bordered table-hover table-sm mb-0">
-        <thead>
-        <tr>
-          <th>Process UUID</th>
-          <th>Created At</th></th>
-          <th>State</th>
-        </tr>
-        </thead>
-        <tbody>
-        <% @processes.each do |process| %>
-          <tr class="<%= dom_class(process) %>" id="<%= dom_id(process) %>">
-            <td><%= process.id %></td>
-            <td><%= relative_time(process.created_at) %></td>
-            <td><%= tag.pre JSON.pretty_generate(process.state) %></td>
+  <% elsif @processes.present? %>
+    <div class="card my-3">
+      <div class="table-responsive">
+        <table class="table card-table table-bordered table-hover table-sm mb-0">
+          <thead>
+          <tr>
+            <th>Process UUID</th>
+            <th>Created At</th></th>
+            <th>State</th>
           </tr>
-        <% end %>
-        </tbody>
-      </table>
+          </thead>
+          <tbody>
+          <% @processes.each do |process| %>
+            <tr class="<%= dom_class(process) %>" id="<%= dom_id(process) %>">
+              <td><%= process.id %></td>
+              <td><%= relative_time(process.created_at) %></td>
+              <td><%= tag.pre JSON.pretty_generate(process.state) %></td>
+            </tr>
+          <% end %>
+          </tbody>
+        </table>
+      </div>
     </div>
-  </div>
-<% else %>
-  <div class="card my-3">
-    <div class="card-body">
-      <p class="card-text">
-        <em>No GoodJob processes found.</em>
-      </p>
+  <% else %>
+    <div class="card my-3">
+      <div class="card-body">
+        <p class="card-text">
+          <em>No GoodJob processes found.</em>
+        </p>
+      </div>
     </div>
-  </div>
-<% end %>
+  <% end %>
+</div>

data/engine/app/views/good_job/shared/_chart.erb

@@ -1,25 +1,3 @@
 <div class="chart-wrapper">
-  <canvas id="chart"></canvas>
+  <canvas class="chart" data-json="<%= chart_data.to_json %>"></canvas>
 </div>
-
-<%= javascript_tag nonce: true do %>
-  const chartData = <%== chart_data.to_json %>;
-
-  const ctx = document.getElementById('chart').getContext('2d');
-  const chart = new Chart(ctx, {
-    type: 'line',
-    data: {
-      labels: chartData.labels,
-      datasets: chartData.datasets
-    },
-    options: {
-      responsive: true,
-      maintainAspectRatio: false,
-      scales: {
-        y: {
-          beginAtZero: true
-        }
-      }
-    }
-  });
-<% end %>

data/engine/app/views/good_job/shared/_filter.erb

@@ -1,4 +1,5 @@
 <%= form_with(url: "", method: :get, local: true, id: "filter_form") do |form| %>
+  <%= hidden_field_tag :poll, value: params[:poll] %>
   <div class="d-flex flex-row w-100">
     <div class="me-2">
       <label for="job_class_filter">Job class</label>
@@ -55,4 +56,4 @@
       });
     })
   })
-<% end %>
+<% end %>

data/engine/app/views/layouts/good_job/base.html.erb

@@ -45,7 +45,10 @@
             </div>
           </li>
         </ul>
-        <div class="text-muted" title="Now is <%= Time.current %>">Times are displayed in <%= Time.current.zone %> timezone</div>
+        <div>
+          <input type="checkbox" id="toggle-poll" name="toggle-poll" data-gj-action='change#togglePoll' <%= 'checked' if params[:poll].present? %>>
+          <label for="toggle-poll">Live Poll</label>
+        </div>
       </div>
     </div>
   </nav>
@@ -70,7 +73,24 @@
         <button type="button" class="btn-close" data-bs-dismiss="alert" aria-label="Close"></button>
       </div>
     <% end %>
+
     <%= yield %>
   </div>
+
+  <footer class="footer mt-auto py-3 bg-light fixed-bottom" id="footer" data-gj-poll-replace>
+    <div class="container-fluid">
+      <div class="row">
+        <div class="col-6">
+          <span class="text-muted">
+            Last updated: <time id="page-updated-at" datetime="<%= Time.current.utc.iso8601 %>"><%= Time.current %></time>
+          </span>
+        </div>
+
+        <div class="col-6 text-end">
+          Remember, you're doing a Good Job too!
+        </div>
+      </div>
+    </div>
+  </footer>
 </body>
 </html>

data/lib/good_job/active_job_job.rb

@@ -15,7 +15,14 @@ module GoodJob
     # Attached to a Job's Execution when the Job is discarded.
     DiscardJobError = Class.new(StandardError)
 
-    self.table_name = 'good_jobs'
+    class << self
+      delegate :table_name, to: Execution
+
+      def table_name=(_value)
+        raise NotImplementedError, 'Assign GoodJob::Execution.table_name directly'
+      end
+    end
+
     self.primary_key = 'active_job_id'
     self.advisory_lockable_column = 'active_job_id'
 

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '2.10.0'
+  VERSION = '2.11.2'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 2.10.0
+  version: 2.11.2
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-02-18 00:00:00.000000000 Z
+date: 2022-03-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob