good_job 3.0.0 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a4f4e14bae83760bdeb35557a727116d053e47a9b7dbf501914606dbc45c2ef7
-  data.tar.gz: c06e9165b7ac5d87af0ab75291fedd0194018752f34332c1a16930ddca53e3dd
+  metadata.gz: bc6e3cffa2189a101181b7ceb278de97ded27d14dd6da915ea8fe3729907dd88
+  data.tar.gz: 0504d2f2efd09d1d52669597e364f62b69edc739d3c76b8bcfdd1933ea7cb8da
 SHA512:
-  metadata.gz: 949cfcf1506ff658666113e035a875ed4ab44fa3e2c674e4d5d1440bf484d39793a1ee595ae0cb6f719136ce5541aae09a7fc2d66a049ea1bcf973b0165dc8a4
-  data.tar.gz: 9c9ff56251e01d60823a50f4edcea74034a36ef93929cf59adc588273da86d1e04f67c1b25d7d35efecda9309acf7f49f81d29fbc7f94249be0e3732087f9081
+  metadata.gz: c10f675a3635f481e370502828ed0902aff3464e19f161bf5f300463a2671ecac09643eb0943030eda30163171b67986b0f9e31ca423ad5da9e4ffd5b23a7df7
+  data.tar.gz: 3ab12e91adb910040a4007140054cc048a0f83e34e54ee649e52ea673a27b95cfb267b56fae98a7605616e4c3f55d17af403c99bf9ba2a5d504dffdd917a7cbb

data/CHANGELOG.md

@@ -1,5 +1,67 @@
 # Changelog
 
+## [v3.1.0](https://github.com/bensheldon/good_job/tree/v3.1.0) (2022-07-11)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.0.2...v3.1.0)
+
+**Implemented enhancements:**
+
+- Improve Dashboard display of parameters \(CronEntry kwargs; Process configuration; Job and Execution database values\) [\#662](https://github.com/bensheldon/good_job/pull/662) ([bensheldon](https://github.com/bensheldon))
+
+**Fixed bugs:**
+
+- Don't delegate `GoodJob::Job#status` to executions to avoid race condition [\#661](https://github.com/bensheldon/good_job/pull/661) ([bensheldon](https://github.com/bensheldon))
+
+**Closed issues:**
+
+- How to suppress repetitive logs in development? [\#658](https://github.com/bensheldon/good_job/issues/658)
+- 500 Internal Server Error Exception in web interface trying to view running jobs [\#656](https://github.com/bensheldon/good_job/issues/656)
+- Cron schedule page in dashboard not showing kwargs [\#608](https://github.com/bensheldon/good_job/issues/608)
+- Paralelism x database connections [\#569](https://github.com/bensheldon/good_job/issues/569)
+
+**Merged pull requests:**
+
+- Show job/cron/process counts in the Navbar [\#663](https://github.com/bensheldon/good_job/pull/663) ([bensheldon](https://github.com/bensheldon))
+- Dequeing should be first-in first-out [\#651](https://github.com/bensheldon/good_job/pull/651) ([jrochkind](https://github.com/jrochkind))
+
+## [v3.0.2](https://github.com/bensheldon/good_job/tree/v3.0.2) (2022-07-10)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.0.1...v3.0.2)
+
+**Fixed bugs:**
+
+- Copy forward concurrency key value when retrying a job, rather than regenerating it [\#622](https://github.com/bensheldon/good_job/issues/622)
+- All concurrency controlled jobs throw exceptions and are rescheduled if they are called using perform\_now [\#591](https://github.com/bensheldon/good_job/issues/591)
+
+**Closed issues:**
+
+- Queue config not respecting limits [\#659](https://github.com/bensheldon/good_job/issues/659)
+- UI engine does not work without explicit require [\#646](https://github.com/bensheldon/good_job/issues/646)
+- Should `:inline` adapter mode retry jobs? [\#611](https://github.com/bensheldon/good_job/issues/611)
+- Error Job Not Preserved  [\#594](https://github.com/bensheldon/good_job/issues/594)
+- Jobs never get run... [\#516](https://github.com/bensheldon/good_job/issues/516)
+- Release GoodJob 3.0 [\#507](https://github.com/bensheldon/good_job/issues/507)
+- Improve security of Gem releases [\#422](https://github.com/bensheldon/good_job/issues/422)
+
+**Merged pull requests:**
+
+- Preserve initial concurrency key when retrying jobs [\#657](https://github.com/bensheldon/good_job/pull/657) ([bensheldon](https://github.com/bensheldon))
+- Add Dashboard troubleshooting note to explicitly require the engine [\#654](https://github.com/bensheldon/good_job/pull/654) ([bensheldon](https://github.com/bensheldon))
+- Removes wrong parentheses [\#653](https://github.com/bensheldon/good_job/pull/653) ([esasse](https://github.com/esasse))
+
+## [v3.0.1](https://github.com/bensheldon/good_job/tree/v3.0.1) (2022-07-02)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.0.0...v3.0.1)
+
+**Closed issues:**
+
+- ERROR:  relation "good\_jobs" does not exist at character 454 [\#308](https://github.com/bensheldon/good_job/issues/308)
+
+**Merged pull requests:**
+
+- Fix `GoodJob.cleanup_preserved_jobs` to use `delete_all` instead of `destroy_all` [\#652](https://github.com/bensheldon/good_job/pull/652) ([bensheldon](https://github.com/bensheldon))
+- Create codeql-analysis.yml [\#648](https://github.com/bensheldon/good_job/pull/648) ([bensheldon](https://github.com/bensheldon))
+
 ## [v3.0.0](https://github.com/bensheldon/good_job/tree/v3.0.0) (2022-06-26)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.99.0...v3.0.0)
@@ -1011,7 +1073,6 @@
 
 **Closed issues:**
 
-- ERROR:  relation "good\_jobs" does not exist at character 454 [\#308](https://github.com/bensheldon/good_job/issues/308)
 - Add Frozen String Literal to all files [\#298](https://github.com/bensheldon/good_job/issues/298)
 - Support for good\_job without Rails? [\#295](https://github.com/bensheldon/good_job/issues/295)
 

data/README.md

@@ -370,6 +370,16 @@ GoodJob includes a Dashboard as a mountable `Rails::Engine`.
 
     See more at [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
 
+**Troubleshooting the Dashboard:** Some applications are unable to autoload the Goodjob Engine. To work around this, explicitly require the Engine at the top of your `config/application.rb` file, immediately after Rails is required and before Bundler requires the Rails' groups.
+
+```ruby
+# config/application.rb
+require_relative 'boot'
+require 'rails/all'
+require 'good_job/engine' # <= Add this line
+# ...
+```
+
 #### API-only Rails applications
 
 API-only Rails applications may not have all of the required Rack middleware for the GoodJob Dashboard to function. To re-add the middlware:
@@ -713,7 +723,7 @@ Each GoodJob execution thread requires its own database connection that is autom
 
 ```yaml
 # config/database.yml
-pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + 3 + (ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i %>
+pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5).to_i + 3 + ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i %>
 ```
 
 To calculate the total number of the database connections you'll need:

data/app/controllers/good_job/jobs_controller.rb

@@ -56,7 +56,7 @@ module GoodJob
     end
 
     def show
-      @job = Job.find(params[:id])
+      @job = Job.includes_advisory_locks.find(params[:id])
     end
 
     def discard

data/app/views/good_job/jobs/_executions.erb

@@ -38,7 +38,7 @@
           </div>
         <% end %>
         <div>
-          <%= tag.pre JSON.pretty_generate(execution.serialized_params), id: dom_id(execution, "params"), class: "collapse bg-light card card-body p-3 my-3" %>
+          <%= tag.pre JSON.pretty_generate(execution.display_serialized_params), id: dom_id(execution, "params"), class: "collapse bg-light card card-body p-3 my-3" %>
         </div>
       <% end %>
     <% end %>

data/app/views/good_job/jobs/_table.erb

@@ -13,7 +13,7 @@
             <th>Executions</th>
             <th>Error</th>
             <th>
-              ActiveJob Params&nbsp;
+              Parameters&nbsp;
               <%= tag.button "Toggle", type: "button", class: "btn btn-sm btn-outline-primary", role: "button",
                              data: { bs_toggle: "collapse", bs_target: ".job-params" },
                              aria: { expanded: false, controls: jobs.map { |job| "##{dom_id(job, "params")}" }.join(" ") }
@@ -71,7 +71,7 @@
                                  data: { bs_toggle: "collapse", bs_target: "##{dom_id(job, 'params')}" },
                                  aria: { expanded: false, controls: dom_id(job, "params") }
                   %>
-                  <%= tag.pre JSON.pretty_generate(job.serialized_params), id: dom_id(job, "params"), class: "collapse job-params" %>
+                  <%= tag.pre JSON.pretty_generate(job.display_serialized_params), id: dom_id(job, "params"), class: "collapse job-params" %>
                 </td>
                 <td>
                   <div class="text-nowrap">

data/app/views/good_job/jobs/show.html.erb

@@ -54,4 +54,4 @@
   <%= tag.pre @job.serialized_params["arguments"].map(&:inspect).join(', ') %>
 </div>
 
-<%= render 'executions', executions: @job.executions.reverse %>
+<%= render 'executions', executions: @job.executions.includes_advisory_locks.reverse %>

data/app/views/good_job/shared/_filter.erb

@@ -11,7 +11,7 @@
         <li class="nav-item">
           <%= link_to url_for({state: name}), class: "nav-link #{"active" if params[:state] == name}" do %>
             <%= t(name, scope: 'good_job.status') %>
-            <span class="badge bg-primary rounded-pill <%= "bg-secondary" if count == 0 %>"><%= count %></span>
+            <span class="badge bg-primary rounded-pill <%= "bg-secondary" if count == 0 %>"><%= number_with_delimiter(count) %></span>
           <% end %>
         </li>
       <% end %>

data/app/views/good_job/shared/_navbar.erb

@@ -8,13 +8,25 @@
     <div class="collapse navbar-collapse" id="navbarSupportedContent">
       <ul class="navbar-nav me-auto">
         <li class="nav-item">
-          <%= link_to t(".jobs"), jobs_path, class: ["nav-link", ("active" if controller_name == 'jobs')] %>
+          <%= link_to jobs_path, class: ["nav-link", ("active" if controller_name == 'jobs')] do %>
+            <%= t(".jobs") %>
+            <% jobs_count = GoodJob::Job.count %>
+            <span class="badge bg-secondary rounded-pill"><%= number_to_human(jobs_count) %></span>
+        <% end %>
         </li>
         <li class="nav-item">
-          <%= link_to t(".cron_schedules"), cron_entries_path, class: ["nav-link", ("active" if controller_name == 'cron_entries')] %>
+          <%= link_to cron_entries_path, class: ["nav-link", ("active" if controller_name == 'cron_entries')] do %>
+            <%= t(".cron_schedules") %>
+            <% cron_entries_count = GoodJob::CronEntry.all.size %>
+            <span class="badge bg-secondary rounded-pill"><%= cron_entries_count %></span>
+        <% end %>
         </li>
         <li class="nav-item">
-          <%= link_to t(".processes"), processes_path, class: ["nav-link", ("active" if controller_name == 'processes')] %>
+          <%= link_to processes_path, class: ["nav-link", ("active" if controller_name == 'processes')] do %>
+            <%= t(".processes") %>
+            <% processes_count = GoodJob::Process.count %>
+            <span class="badge bg-secondary rounded-pill <%= "bg-danger" if processes_count == 0 %>"><%= processes_count %></span>
+          <% end %>
         </li>
       </ul>
       <div class="nav-item pe-2">

data/config/locales/en.yml

@@ -63,3 +63,22 @@ en:
       retried: Retried
       running: Running
       scheduled: Scheduled
+  number:
+    format:
+      delimiter: ","
+      separator: "."
+    human:
+      decimal_units:
+        format: "%n%u"
+        units:
+          billion: B
+          hundred: ''
+          million: M
+          quadrillion: Q
+          thousand: K
+          trillion: T
+          unit: ''
+      format:
+        delimiter: ","
+        precision: 3
+        separator: "."

data/config/locales/es.yml

@@ -63,3 +63,22 @@ es:
       retried: reintentado
       running: Corriendo
       scheduled: Programado
+  number:
+    format:
+      delimiter: " "
+      separator: ","
+    human:
+      decimal_units:
+        format: "%n%u"
+        units:
+          billion: B
+          hundred: ''
+          million: M
+          quadrillion: q
+          thousand: k
+          trillion: T
+          unit: ''
+      format:
+        delimiter: " "
+        precision: 3
+        separator: ","

data/config/locales/nl.yml

@@ -63,3 +63,22 @@ nl:
       retried: Opnieuw geprobeerd
       running: Rennen
       scheduled: Gepland
+  number:
+    format:
+      delimiter: "."
+      separator: ","
+    human:
+      decimal_units:
+        format: "%n%u"
+        units:
+          billion: B
+          hundred: ''
+          million: M
+          quadrillion: Q
+          thousand: K
+          trillion: T
+          unit: ''
+      format:
+        delimiter: "."
+        precision: 3
+        separator: ","

data/config/locales/ru.yml

@@ -87,3 +87,22 @@ ru:
       retried: Повторная попытка
       running: Бег
       scheduled: по расписанию
+  number:
+    format:
+      delimiter: " "
+      separator: ","
+    human:
+      decimal_units:
+        format: "%n%u"
+        units:
+          billion: Б
+          hundred: ''
+          million: М
+          quadrillion: Q
+          thousand: К
+          trillion: Т
+          unit: ''
+      format:
+        delimiter: " "
+        precision: 3
+        separator: ","

data/lib/good_job/active_job_extensions/concurrency.rb

@@ -10,8 +10,18 @@ module GoodJob
         end
       end
 
+      module Prepends
+        def deserialize(job_data)
+          super
+          self.good_job_concurrency_key = job_data['good_job_concurrency_key']
+        end
+      end
+
       included do
+        prepend Prepends
+
         class_attribute :good_job_concurrency_config, instance_accessor: false, default: {}
+        attr_writer :good_job_concurrency_key
 
         around_enqueue do |job, block|
           # Don't attempt to enforce concurrency limits with other queue adapters.
@@ -27,6 +37,8 @@ module GoodJob
                       (total_limit.present? && (0...Float::INFINITY).cover?(total_limit))
           next(block.call) unless has_limit
 
+          # Only generate the concurrency key on the initial enqueue in case it is dynamic
+          job.good_job_concurrency_key ||= job._good_job_concurrency_key
           key = job.good_job_concurrency_key
           next(block.call) if key.blank?
 
@@ -80,7 +92,15 @@ module GoodJob
         end
       end
 
+      # Existing or dynamically generated concurrency key
+      # @return [Object] concurrency key
       def good_job_concurrency_key
+        @good_job_concurrency_key || _good_job_concurrency_key
+      end
+
+      # Generates the concurrency key from the configuration
+      # @return [Object] concurrency key
+      def _good_job_concurrency_key
         key = self.class.good_job_concurrency_config[:key]
         return if key.blank?
 

data/lib/good_job/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module GoodJob
   # GoodJob gem version.
-  VERSION = '3.0.0'
+  VERSION = '3.1.0'
 end

data/lib/good_job.rb

@@ -145,7 +145,7 @@ module GoodJob
       old_jobs = old_jobs.not_discarded unless include_discarded
       old_jobs_count = old_jobs.count
 
-      GoodJob::Execution.where(job: old_jobs).destroy_all
+      GoodJob::Execution.where(job: old_jobs).delete_all
       payload[:destroyed_records_count] = old_jobs_count
     end
   end

data/lib/models/good_job/cron_entry.rb

@@ -112,9 +112,11 @@ module GoodJob # :nodoc:
         class: job_class,
         cron: schedule,
         set: display_property(set),
-        args: display_property(args),
         description: display_property(description),
-      }
+      }.tap do |properties|
+        properties[:args] = display_property(args) if args.present?
+        properties[:kwargs] = display_property(kwargs) if kwargs.present?
+      end
     end
 
     private

data/lib/models/good_job/execution.rb

@@ -88,6 +88,18 @@ module GoodJob
     # @return [ActiveRecord::Relation]
     scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
 
+    # Order jobs by created_at, for first-in first-out
+    # @!method creation_ordered
+    # @!scope class
+    # @return [ActiveRecord:Relation]
+    scope :creation_ordered, -> { order('created_at ASC') }
+
+    # Order jobs for de-queueing
+    # @!method dequeue_ordered
+    # @!scope class
+    # @return [ActiveRecord:Relation]
+    scope :dequeue_ordered, -> { priority_ordered.creation_ordered }
+
     # Order jobs by scheduled or created (oldest first).
     # @!method schedule_ordered
     # @!scope class
@@ -154,7 +166,7 @@ module GoodJob
     #   raised, if any (if the job raised, then the second array entry will be
     #   +nil+). If there were no jobs to execute, returns +nil+.
     def self.perform_with_advisory_lock
-      unfinished.priority_ordered.only_scheduled.limit(1).with_advisory_lock(unlock_session: true) do |executions|
+      unfinished.dequeue_ordered.only_scheduled.limit(1).with_advisory_lock(unlock_session: true) do |executions|
         execution = executions.first
         break if execution.blank?
         break :unlocked unless execution&.executable?
@@ -279,7 +291,9 @@ module GoodJob
     # @return [Symbol]
     def status
       if finished_at.present?
-        if error.present?
+        if error.present? && retried_good_job_id.present?
+          :retried
+        elsif error.present? && retried_good_job_id.nil?
           :discarded
         else
           :finished
@@ -297,8 +311,20 @@ module GoodJob
       end
     end
 
+    # Return formatted serialized_params for display in the dashboard
+    # @return [Hash]
+    def display_serialized_params
+      serialized_params.merge({
+                                _good_job: attributes.except('serialized_params', 'locktype', 'owns_advisory_lock'),
+                              })
+    end
+
     def running?
-      performed_at? && !finished_at?
+      if has_attribute?(:locktype)
+        self['locktype'].present?
+      else
+        advisory_locked?
+      end
     end
 
     def number
@@ -314,7 +340,7 @@ module GoodJob
     def queue_latency
       now = Time.zone.now
       expected_start = scheduled_at || created_at
-      actual_start = performed_at || now
+      actual_start = performed_at || finished_at || now
 
       actual_start - expected_start unless expected_start >= now
     end
@@ -330,6 +356,7 @@ module GoodJob
       serialized_params.deep_dup
                        .tap do |job_data|
         job_data["provider_job_id"] = id
+        job_data["good_job_concurrency_key"] = concurrency_key if concurrency_key
       end
     end
 

data/lib/models/good_job/job.rb

@@ -3,7 +3,8 @@ module GoodJob
   # ActiveRecord model that represents an +ActiveJob+ job.
   # There is not a table in the database whose discrete rows represents "Jobs".
   # The +good_jobs+ table is a table of individual {GoodJob::Execution}s that share the same +active_job_id+.
-  # A single row from the +good_jobs+ table of executions is fetched to represent an Job
+  # A single row from the +good_jobs+ table of executions is fetched to represent a Job.
+  #
   class Job < BaseRecord
     include Filterable
     include Lockable
@@ -72,9 +73,51 @@ module GoodJob
       serialized_params['job_class']
     end
 
-    # The status of the Job, based on the state of its most recent execution.
-    # @return [Symbol]
-    delegate :status, :last_status_at, to: :head_execution
+    def last_status_at
+      finished_at || performed_at || scheduled_at || created_at
+    end
+
+    def status
+      if finished_at.present?
+        if error.present? && retried_good_job_id.present?
+          :retried
+        elsif error.present? && retried_good_job_id.nil?
+          :discarded
+        else
+          :finished
+        end
+      elsif (scheduled_at || created_at) > DateTime.current
+        if serialized_params.fetch('executions', 0) > 1
+          :retried
+        else
+          :scheduled
+        end
+      elsif running?
+        :running
+      else
+        :queued
+      end
+    end
+
+    # Override #reload to add a custom scope to ensure the reloaded record is the head execution
+    # @return [Job]
+    def reload(options = nil)
+      self.class.connection.clear_query_cache
+
+      # override with the `where(retried_good_job_id: nil)` scope
+      override_query = self.class.where(retried_good_job_id: nil)
+      fresh_object =
+        if options && options[:lock]
+          self.class.unscoped { override_query.lock(options[:lock]).find(id) }
+        else
+          self.class.unscoped { override_query.find(id) }
+        end
+
+      @attributes = fresh_object.instance_variable_get(:@attributes)
+      @new_record = false
+      @previously_new_record = false
+      self
+    end
 
     # This job's most recent {Execution}
     # @param reload [Booelan] whether to reload executions
@@ -94,7 +137,7 @@ module GoodJob
     # The number of times this job has been executed, according to ActiveJob's serialized state.
     # @return [Numeric]
     def executions_count
-      aj_count = head_execution.serialized_params.fetch('executions', 0)
+      aj_count = serialized_params.fetch('executions', 0)
       # The execution count within serialized_params is not updated
       # once the underlying execution has been executed.
       if status.in? [:discarded, :finished, :running]
@@ -114,7 +157,15 @@ module GoodJob
     # If the job has been retried, the error will be fetched from the previous {Execution} record.
     # @return [String]
     def recent_error
-      head_execution.error || executions[-2]&.error
+      error || executions[-2]&.error
+    end
+
+    # Return formatted serialized_params for display in the dashboard
+    # @return [Hash]
+    def display_serialized_params
+      serialized_params.merge({
+                                _good_job: attributes.except('serialized_params', 'locktype', 'owns_advisory_lock'),
+                              })
     end
 
     # Tests whether the job is being executed right now.
@@ -192,7 +243,6 @@ module GoodJob
 
         raise ActionForStateMismatchError if execution.finished_at.present?
 
-        execution = head_execution(reload: true)
         execution.update(scheduled_at: scheduled_at)
       end
     end

data/lib/models/good_job/process.rb

@@ -45,6 +45,8 @@ module GoodJob # :nodoc:
         hostname: Socket.gethostname,
         pid: ::Process.pid,
         proctitle: $PROGRAM_NAME,
+        preserve_job_records: GoodJob.preserve_job_records,
+        retry_on_unhandled_error: GoodJob.retry_on_unhandled_error,
         schedulers: GoodJob::Scheduler.instances.map(&:name),
       }
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 3.0.0
+  version: 3.1.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-06-26 00:00:00.000000000 Z
+date: 2022-07-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob