good_job 3.27.4 → 3.28.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa73a0d9ffc1feb2548c525af767537d79a8b6fdf4e03092c4963d91041eb73a
-  data.tar.gz: 41365ce2085788ff587d85a96d6bf1568fb5861ea025e98c87990ae65485ffc8
+  metadata.gz: b22f482dcf118c82d0b5889c6c37fd72bb65ff9a010f4fcc486bc2f0c4fc0bbe
+  data.tar.gz: 83b1d597eee0f0c9c37b4afdf82f96e2164bef2d3b27181c10678327803f718e
 SHA512:
-  metadata.gz: a300b98ed92353715dff14d18940f0e9fa2896e1cda56428c026e62b162e3f7a2f5a07eba2a9501af2d0c1f4fa51f3c225a4ecc103f73e8eac862ec343902e6e
-  data.tar.gz: 25241155b1d5e375a903cbeec98e88ecaaca2fc46b1134c92ed400fbf2b024a637744db072fe42fb9af486e7ce43cec5262c6c31adceb2f948af3aacb7cc9584
+  metadata.gz: b0b84290323a36b818cf92763e9df745737099f3990c934914e8ae700d6cb3a2ceb237764a8ff24fe923d5a8af0d18394e14a6aec516085a2bf057579341747e
+  data.tar.gz: 030e271e5218499dbbce39e21803a5da6ede65422addb33ba5a0024582ac0996418471664061ce54700b1e9a5819eeeac517c479bb79d329577985e148752811

data/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## [v3.28.0](https://github.com/bensheldon/good_job/tree/v3.28.0) (2024-04-19)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.27.4...v3.28.0)
+
+**Implemented enhancements:**
+
+- Store and optionally display the full execution backtrace [\#1328](https://github.com/bensheldon/good_job/pull/1328) ([Earlopain](https://github.com/Earlopain))
+- Store error backtraces on discrete executions [\#1325](https://github.com/bensheldon/good_job/pull/1325) ([Intrepidd](https://github.com/Intrepidd))
+
+**Fixed bugs:**
+
+- add missing dropdown-item class [\#1327](https://github.com/bensheldon/good_job/pull/1327) ([patriciomacadden](https://github.com/patriciomacadden))
+
+**Closed issues:**
+
+- Proposal: Migrating Documentation to a Separate Website [\#1324](https://github.com/bensheldon/good_job/issues/1324)
+- Potential documentation error in GoodJob::ActiveJobExtensions::NotifyOptions [\#1321](https://github.com/bensheldon/good_job/issues/1321)
+- ActiveSupport::CurrentAttributes reset after `perform_later` [\#1320](https://github.com/bensheldon/good_job/issues/1320)
+- Storing backtrace in database? [\#1162](https://github.com/bensheldon/good_job/issues/1162)
+- Potential locking race condition when using cron scheduler across multiple processes [\#731](https://github.com/bensheldon/good_job/issues/731)
+
+**Merged pull requests:**
+
+- docs: corrected a typo regarding the use of GoodJob::ActiveJobExtensions::NotifyOptions [\#1322](https://github.com/bensheldon/good_job/pull/1322) ([pgvsalamander](https://github.com/pgvsalamander))
+- Add "best practices" section to Readme [\#1318](https://github.com/bensheldon/good_job/pull/1318) ([bensheldon](https://github.com/bensheldon))
+- Change ApplicationRecord to ApplicationJob for label documentation. [\#1317](https://github.com/bensheldon/good_job/pull/1317) ([frans-k](https://github.com/frans-k))
+- Run test matrix against Ruby 3.3; remove pry [\#1315](https://github.com/bensheldon/good_job/pull/1315) ([bensheldon](https://github.com/bensheldon))
+- Add `Rails.application.load_server` to Demo `config.ru`; quiet puma web-concurrency warnings [\#1314](https://github.com/bensheldon/good_job/pull/1314) ([bensheldon](https://github.com/bensheldon))
+- Fix test leakage of configuration double [\#1312](https://github.com/bensheldon/good_job/pull/1312) ([bensheldon](https://github.com/bensheldon))
+- Rewrite queries to all use bind parameters and prepare: true [\#1308](https://github.com/bensheldon/good_job/pull/1308) ([bensheldon](https://github.com/bensheldon))
+
 ## [v3.27.4](https://github.com/bensheldon/good_job/tree/v3.27.4) (2024-04-04)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v3.27.3...v3.27.4)

data/README.md

@@ -72,6 +72,11 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
     - [Write tests](#write-tests)
     - [PgBouncer compatibility](#pgbouncer-compatibility)
     - [CLI HTTP health check probes](#cli-http-health-check-probes)
+- [Doing your best job with GoodJob](#doing-your-best-job-with-goodjob)
+    - [Sizing jobs: mice and elephants](#sizing-jobs-mice-and-elephants)
+    - [Isolating by total latency](#isolating-by-total-latency)
+    - [Configuring your queues](#configuring-your-queues)
+    - [Additional observations](#additional-observations)
 - [Contribute](#contribute)
     - [Gem development](#gem-development)
         - [Development setup](#development-setup)
@@ -500,12 +505,12 @@ Higher priority numbers run first in all versions of GoodJob v3.x and below. Goo
 Labels are the recommended way to add context or metadata to specific jobs. For example, all jobs that have a dependency on an email service could be labeled `email`. Using labels requires adding the Active Job extension `GoodJob::ActiveJobExtensions::Labels` to your job class.
 
 ```ruby
-class ApplicationRecord < ActiveJob::Base
+class ApplicationJob < ActiveJob::Base
   include GoodJob::ActiveJobExtensions::Labels
 end
 
 # Add a default label to every job within the class
-class WelcomeJob < ApplicationRecord
+class WelcomeJob < ApplicationJob
   self.good_job_labels = ["email"]
 
   def perform
@@ -1527,6 +1532,90 @@ gem 'webrick'
 
 If WEBrick is configured to be used, but the dependency is not found, GoodJob will log a warning and fallback to the default probe server.
 
+## Doing your best job with GoodJob
+
+_This section explains how to use GoodJob the most efficiently and performantly, according to its maintainers. GoodJob is very flexible and you don’t necessarily have to use it this way, but the concepts explained here are part of GoodJob’s design intent._
+
+Background jobs are hard. There are two extremes:
+
+- **Throw resources (compute, servers, money) at it** by creating dedicated processes (or servers) for each type of job or queue and scaling them independently to achieve the lowest latency and highest throughput.
+- **Do the best you can in a small budget** by creating dedicated _thread pools_ within a process for each type of job or queue to produce quality-of-service and compromise maximum latency (or tail latency) because of shared resources and thread contention. You can even run them in the web process if you’re really cheap.
+
+This section will largely focused on optimizing within the latter small-budget scenario, but the concepts and explanation should help you optimize the big-budget scenario too.
+
+Let’s start with anti-patterns, and then the rest of this section will explain an alternative:
+
+- **Don’t use functional names for your queues** like `mailers` or `sms` or `turbo` or `batch`.  Instead name them after the total latency target (the total duration within queue and executing till finish) you expect for that job e.g.`latency_30s` or `latency_5m` or `literally_whenever`.
+- **Priority can’t fix a lack of capacity.** Priority rules (i.e. weighing or ordering which jobs or queues execute first) only works when there is  capacity available to execute that _next_ job. When all capacity is in-use, priority cannot preempt a job that is already executing ("head-of-line blocking").
+
+The following will explain methods to create homogenous workloads (based on latency) and increase execution capacity when queuing latency causes the jobs to exceed their total latency target.
+
+### Sizing jobs: mice and elephants
+
+Queuing theory will refer to fast/small/low-latency tasks as **Mice** (e.g. a password reset email, an MFA token via SMS) and slow/big/high-latency tasks as **Elephants** (e.g. sending an email newsletter to 10k recipients, a batched update that touches every record in the database).
+
+Explicitly group your jobs by their latency: how quickly you expect them to finish to achieve your expected quality of service. This should be their **total latency** (or duration) which is the sum of: **queuing latency** which is how long the job waits in queue until execution capacity becomes available (which ideally should be zero, because you have idle capacity and can start executing a job immediately as soon as it is enqueued or upon its scheduled time) and **execution latency** which is how long the job’s execution takes (e.g. the email being sent). Example: I expect this Password Reset Email Job to have a total latency of 30 seconds or less.
+
+In a working application, you likely will have more gradations than just small and big or slow and fast (analogously: badgers, wildebeests; maybe even tardigrades or blue whales for tiny and huge, respectively), but there will regardless be a relatively small and countable number of discrete latency buckets to organize your jobs into.
+
+### Isolating by total latency
+
+The most efficient workloads are homogenous (similar) workloads. If you know every job to be executed will take about the same amount of time, you can estimate the maximum delay for a new job at the back of the queue and have that drive decisions about capacity. Alternatively, if those jobs are heterogenous (mixed) it’s possible that a very slow/long-duration job could hold everything back for much longer than anticipated and it’s sorta random. That’s bad!
+
+A fun visual image here for a single-file queue is a doorway: If you only have 1 doorway, it must be big enough to fit an elephant. But if an elephant is going through the door (and it will go through slowly!) no mice can fit through the door until the elephant is fully clear. Your mice will be delayed!
+
+Priority will not help when an elephant is in the doorway. Yes, you could say mice have a higher priority than elephants and always allow any mouse to go _before_ any elephant in queue will start. But once an elephant *has started* going through the door, any subsequent mouse who arrives must wait for the elephant to egress regardless of their priority. In Active Job and Ruby, it’s really hard to stop or cancel or preempt a running job (unless you’ve already architected that into your jobs, like with the [`job-iteration`](https://github.com/Shopify/job-iteration) library)
+
+The best solution is to have a 2nd door, but only sized for mice, so an elephant can’t ever block it. With a mouse-sized doorway _and_ an elephant-sized doorway, mice can still go through the big elephant door when an elephant isn’t using it. Each door has a _maximum_ size (or “latency”) we want it to accept, and smaller is ok, just not larger.
+
+### Configuring your queues
+
+If we wanted to capture the previous 2-door scenario in GoodJob, we’d configure the queues like this;
+
+```ruby
+config.good_job.queues = "mice:1; elephant,mice:1"
+```
+
+This configuration creates two isolated thread pools (separated by a semicolon) each with 1 thread each (the number after the colon). The 2nd thread pool recognizes that both elephants and mice can use that isolated thread pool; if there is an influx of mice, it's possible to use the elephant’s thread pool if an elephant isn't already in progress.
+
+So what if we add an intermediately-sized `badgers` ? In that case, we can make 3 distinct queues:
+
+```ruby
+config.good_job.queues = "mice:1; badgers,mice:1; elephants,badgers,mice:1"
+```
+
+In this case, we make a mouse sized queue, a badger sized queue, and an elephant sized queue. We can simplify this even further:
+
+```ruby
+config.good_job.queues = "mice:1; badgers,mice:1; *:1"
+```
+
+Using the wildcard  `*` for any queue also helps ensure that if a job is enqueued to a newly declared queue (maybe via a dependency or just inadvertently) it will still get executed until you notice and decide on its appropriate latency target.
+
+In these examples, the order doesn’t matter; it just is maybe more readable to go from the lowest-latency to largest-latency pool (the semicolon groups), and then within a pool to list the largest allowable latency first (the commas). Nothing here is about “job priority” or “queue priority”, this is wholly about grouping.
+
+In your application, not the zoo, you’ll want to enqueue your `PaswordResetJob` on the `mice` queue, your `CreateComplicatedObjectJob` on the `badger` queue, and your `AuditEveryAccountEverJob` on the `elephant` queue. But you want to name your queues by latency, so that ends up being:
+
+```ruby
+config.good_job.queues = "latency_30s:1; latency_2m,latency_30s:1; *:1"
+```
+
+And you likely want to have more than one thread (though more than 3-5 threads per process will cause thread contention and slow everything down a bit):
+
+```ruby
+config.good_job.queues = "latency_30s:2; latency_2m,latency_30s:2; *:2"
+```
+
+### Additional observations
+
+- Unlike GoodJob, other Active Job backends may treat a "queue" and an "isolated execution pool" as one and the same. GoodJob allows composing multiple Active Job queues into the same pool for flexibility and to make it easier to migrate from functionally-named queues to latency-based ones.
+- You don't *have* to name your queues explicitly like `latency_30s` but it makes it easier to identify outliers and communicate your operational targets. Many people push back on this; that's ok. An option to capture functional details is to use GoodJob's Labels feature instead of encoding them in the queue name.
+- The downside of organizing your jobs like this is that you may have jobs with the same latency target but wildly different operational parameters, like being coupled to another system that has limited throughput or questionable reliability. GoodJob offers Concurrency and Throttling Controls, but isolation is always the most performant and reliable option, though it requires dedicated resources and costs more.
+- Observe, monitor, and adapt your job queues over time. You likely have incomplete information about the execution latency of your jobs inclusive of all dependencies across all scenarios. You should expect to adjust your queues and grouping over time as you observe their behavior.
+- If you find you have unreliable external dependencies that introduce latency, you may also want to further isolate your jobs based on those dependencies, for example, isolating `latency_10s_email_service` to its own execution pool.
+- Scale on queue latency. Per the previous point in which you do not have complete control over execution latency, you do have control over the queue latency. If queue latency is causing your jobs to miss their total latency target, you must add more capacity (e.g. processes or servers.
+- This is all largely about latency-based queue design. It’s possible to go further and organize by latency _and_ parallelism. For that I recommend Nate Berkopec’s [*Complete Guide to Rails Performance*](https://www.railsspeed.com/) which covers things like Amdahl’s Law.
+
 ## Contribute
 
 <!-- Please keep this section in sync with CONTRIBUTING.md -->

data/app/frontend/good_job/style.css

@@ -42,3 +42,7 @@
 .min-w-auto {
   min-width: auto;
 }
+
+.w-fit-content {
+  width: fit-content
+}

data/app/models/concerns/good_job/advisory_lockable.rb

@@ -46,14 +46,19 @@ module GoodJob
         cte_query = cte_query.limit(select_limit) if select_limit
         cte_type = supports_cte_materialization_specifiers? ? :MATERIALIZED : :""
         composed_cte = Arel::Nodes::As.new(cte_table, Arel::Nodes::UnaryOperation.new(cte_type, cte_query.arel))
+
+        lock_condition = "#{function}(('x' || substr(md5(#{connection.quote(table_name)} || '-' || #{connection.quote_table_name(cte_table.name)}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64)::bigint)"
         query = cte_table.project(cte_table[:id])
                          .with(composed_cte)
-                         .where(Arel.sql("#{function}(('x' || substr(md5(#{connection.quote(table_name)} || '-' || #{connection.quote_table_name(cte_table.name)}.#{connection.quote_column_name(column)}::text), 1, 16))::bit(64)::bigint)"))
+                         .where(defined?(Arel::Nodes::BoundSqlLiteral) ? Arel::Nodes::BoundSqlLiteral.new(lock_condition, [], {}) : Arel::Nodes::SqlLiteral.new(lock_condition))
 
         limit = original_query.arel.ast.limit
         query.limit = limit.value if limit.present?
 
-        unscoped.where(arel_table[primary_key].in(query)).merge(original_query.only(:order))
+        # Arel.sql and the IN clause prevent this from being preparable
+        # That's why this is manually composed of BoundSqlLiteral's and an InfixOperation
+        # to sidestep anywhere in Arel where the `collector.preparable = false` is set
+        unscoped.where(Arel::Nodes::InfixOperation.new("IN", arel_table[primary_key], query)).merge(original_query.only(:order))
       end)
 
       # Joins the current query with Postgres's +pg_locks+ table (it provides

data/app/models/concerns/good_job/filterable.rb

@@ -17,9 +17,9 @@ module GoodJob
       scope :display_all, (lambda do |after_scheduled_at: nil, after_id: nil|
         query = order(Arel.sql('COALESCE(scheduled_at, created_at) DESC, id DESC'))
         if after_scheduled_at.present? && after_id.present?
-          query = query.where(Arel.sql('(COALESCE(scheduled_at, created_at), id) < (:after_scheduled_at, :after_id)'), after_scheduled_at: after_scheduled_at, after_id: after_id)
+          query = query.where Arel::Nodes::Grouping.new([coalesce_scheduled_at_created_at, arel_table["id"]]).lt(Arel::Nodes::Grouping.new([bind_value('coalesce', after_scheduled_at, ActiveRecord::Type::DateTime), bind_value('id', after_id, ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Uuid)]))
         elsif after_scheduled_at.present?
-          query = query.where(Arel.sql('(COALESCE(scheduled_at, created_at)) < (:after_scheduled_at)'), after_scheduled_at: after_scheduled_at)
+          query = query.where coalesce_scheduled_at_created_at.lt(bind_value('coalesce', after_scheduled_at, ActiveRecord::Type::DateTime))
         end
         query
       end)
@@ -34,6 +34,7 @@ module GoodJob
         query = query.to_s.strip
         next if query.blank?
 
+        # TODO: turn this into proper bind parameters in Arel
         tsvector = "(to_tsvector('english', id::text) || to_tsvector('english', COALESCE(active_job_id::text, '')) || to_tsvector('english', serialized_params) || to_tsvector('english', COALESCE(error, ''))#{" || to_tsvector('english', COALESCE(array_to_string(labels, ' '), ''))" if labels_migrated?})"
         to_tsquery_function = database_supports_websearch_to_tsquery? ? 'websearch_to_tsquery' : 'plainto_tsquery'
         where("#{tsvector} @@ #{to_tsquery_function}(?)", query)

data/app/models/good_job/base_record.rb

@@ -40,6 +40,10 @@ module GoodJob
       end
     end
 
+    def self.bind_value(name, value, type_class)
+      Arel::Nodes::BindParam.new(ActiveRecord::Relation::QueryAttribute.new(name, value, type_class.new))
+    end
+
     ActiveSupport.run_load_hooks(:good_job_base_record, self)
   end
 end

data/app/models/good_job/batch_record.rb

@@ -18,6 +18,8 @@ module GoodJob
     scope :not_discarded, -> { where(discarded_at: nil) }
     scope :succeeded, -> { finished.not_discarded }
 
+    scope :finished_before, ->(timestamp) { where(arel_table['finished_at'].lteq(bind_value('finished_at', timestamp, ActiveRecord::Type::DateTime))) }
+
     alias_attribute :enqueued?, :enqueued_at
     alias_attribute :discarded?, :discarded_at
     alias_attribute :finished?, :finished_at
@@ -25,9 +27,13 @@ module GoodJob
     scope :display_all, (lambda do |after_created_at: nil, after_id: nil|
       query = order(created_at: :desc, id: :desc)
       if after_created_at.present? && after_id.present?
-        query = query.where(Arel.sql('(created_at, id) < (:after_created_at, :after_id)'), after_created_at: after_created_at, after_id: after_id)
+        query = if Gem::Version.new(Rails.version) < Gem::Version.new('7.0.0.a') || Concurrent.on_jruby?
+                  query.where(Arel.sql('(created_at, id) < (:after_created_at, :after_id)'), after_created_at: after_created_at, after_id: after_id)
+                else
+                  query.where Arel::Nodes::Grouping.new([arel_table["created_at"], arel_table["id"]]).lt(Arel::Nodes::Grouping.new([bind_value('created_at', after_created_at, ActiveRecord::Type::DateTime), bind_value('id', after_id, ActiveRecord::Type::String)]))
+                end
       elsif after_created_at.present?
-        query = query.where(Arel.sql('(after_created_at) < (:after_created_at)'), after_created_at: after_created_at)
+        query = query.where arel_table["created_at"].lt(bind_value('created_at', after_created_at, ActiveRecord::Type::DateTime))
       end
       query
     end)

data/app/models/good_job/discrete_execution.rb

@@ -21,6 +21,13 @@ module GoodJob # :nodoc:
       false
     end
 
+    def self.backtrace_migrated?
+      return true if columns_hash["error_backtrace"].present?
+
+      migration_pending_warning!
+      false
+    end
+
     def number
       serialized_params.fetch('executions', 0) + 1
     end
@@ -58,5 +65,9 @@ module GoodJob # :nodoc:
                                 _good_job_execution: attributes.except('serialized_params'),
                               })
     end
+
+    def filtered_error_backtrace
+      Rails.backtrace_cleaner.clean(error_backtrace || [])
+    end
   end
 end

data/app/models/good_job/execution.rb

@@ -95,7 +95,7 @@ module GoodJob
     # @!method only_scheduled
     # @!scope class
     # @return [ActiveRecord::Relation]
-    scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Arel::Nodes::BindParam.new(ActiveModel::Attribute.with_cast_value("scheduled_at", Time.current, ActiveModel::Type::DateTime.new)))).or(where(scheduled_at: nil)) }
+    scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(bind_value('scheduled_at', DateTime.current, ActiveRecord::Type::DateTime))).or(where(scheduled_at: nil)) }
 
     # Order executions by priority (highest priority first).
     # @!method priority_ordered
@@ -161,7 +161,7 @@ module GoodJob
     # @param timestamp (Float)
     #   Get jobs that finished before this time (in epoch time).
     # @return [ActiveRecord::Relation]
-    scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
+    scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(bind_value('finished_at', timestamp, ActiveRecord::Type::DateTime))) : where.not(finished_at: nil) }
 
     # Get Jobs that started but not finished yet.
     # @!method running
@@ -290,11 +290,12 @@ module GoodJob
       query = advisory_unlocked.unfinished.schedule_ordered
 
       after ||= Time.current
-      after_query = query.where('scheduled_at > ?', after).or query.where(scheduled_at: nil).where('created_at > ?', after)
+      after_bind = bind_value('scheduled_at', after, ActiveRecord::Type::DateTime)
+      after_query = query.where(arel_table['scheduled_at'].gt(after_bind)).or query.where(scheduled_at: nil).where(arel_table['created_at'].gt(after_bind))
       after_at = after_query.limit(limit).pluck(:scheduled_at, :created_at).map { |timestamps| timestamps.compact.first }
 
       if now_limit&.positive?
-        now_query = query.where('scheduled_at < ?', Time.current).or query.where(scheduled_at: nil)
+        now_query = query.where(arel_table['scheduled_at'].lt(bind_value('scheduled_at', Time.current, ActiveRecord::Type::DateTime))).or query.where(scheduled_at: nil)
         now_at = now_query.limit(now_limit).pluck(:scheduled_at, :created_at).map { |timestamps| timestamps.compact.first }
       end
 
@@ -460,6 +461,7 @@ module GoodJob
           if discrete_execution
             discrete_execution.error = error_string
             discrete_execution.error_event = result.error_event
+            discrete_execution.error_backtrace = job_error.backtrace if discrete_execution.class.backtrace_migrated?
           end
         else
           job_attributes[:error] = nil

data/app/models/good_job/job.rb

@@ -44,14 +44,14 @@ module GoodJob
     # @!scope class
     # @param timestamp (DateTime, Time)
     # @return [ActiveRecord::Relation]
-    scope :finished_before, ->(timestamp) { where(arel_table['finished_at'].lteq(timestamp)) }
+    scope :finished_before, ->(timestamp) { where(arel_table['finished_at'].lteq(bind_value('finished_at', timestamp, ActiveRecord::Type::DateTime))) }
 
     # First execution will run in the future
-    scope :scheduled, -> { where(finished_at: nil).where(coalesce_scheduled_at_created_at.gt(DateTime.current)).where(params_execution_count.lt(2)) }
+    scope :scheduled, -> { where(finished_at: nil).where(coalesce_scheduled_at_created_at.gt(bind_value('coalesce', Time.current, ActiveRecord::Type::DateTime))).where(params_execution_count.lt(2)) }
     # Execution errored, will run in the future
-    scope :retried, -> { where(finished_at: nil).where(coalesce_scheduled_at_created_at.gt(DateTime.current)).where(params_execution_count.gt(1)) }
+    scope :retried, -> { where(finished_at: nil).where(coalesce_scheduled_at_created_at.gt(bind_value('coalesce', Time.current, ActiveRecord::Type::DateTime))).where(params_execution_count.gt(1)) }
     # Immediate/Scheduled time to run has passed, waiting for an available thread run
-    scope :queued, -> { where(finished_at: nil).where(coalesce_scheduled_at_created_at.lteq(DateTime.current)).joins_advisory_locks.where(pg_locks: { locktype: nil }) }
+    scope :queued, -> { where(finished_at: nil).where(coalesce_scheduled_at_created_at.lteq(bind_value('coalesce', Time.current, ActiveRecord::Type::DateTime))).joins_advisory_locks.where(pg_locks: { locktype: nil }) }
     # Advisory locked and executing
     scope :running, -> { where(finished_at: nil).joins_advisory_locks.where.not(pg_locks: { locktype: nil }) }
     # Finished executing (succeeded or discarded)

data/app/views/good_job/jobs/_executions.erb

@@ -35,9 +35,35 @@
         </div>
         <% if execution.error %>
           <div class="mt-3 small">
-            <strong class="small"><%=t "good_job.shared.error" %>:</strong>
+            <strong class="small text-danger"><%=t "good_job.shared.error" %>:</strong>
             <code class="text-wrap text-break m-0 text-secondary-emphasis"><%= execution.error %></code>
           </div>
+          <% if GoodJob::DiscreteExecution.backtrace_migrated? && execution.error_backtrace&.any? %>
+            <%= tag.ul class: "nav nav-tabs small w-fit-content", id: dom_id(execution, :tab), role:"tablist" do %>
+              <li class="nav-item" role="presentation">
+                <%= tag.button t(".application_trace"), class: "nav-link active p-1", id: dom_id(execution, :application), data: { bs_toggle: "tab", bs_target: dom_id(execution, :"#application_pane") }, type: "button", role: "tab", aria: { controls: dom_id(execution, :application_pane), selected: true  }  %>
+              </li>
+              <li class="nav-item" role="presentation">
+                <%= tag.button t(".full_trace"), class: "nav-link p-1", id: dom_id(execution, :full), data: { bs_toggle: "tab", bs_target: dom_id(execution, :"#full_pane") }, type: "button", role: "tab", aria: { controls: dom_id(execution, :full_pane), selected: false } %>
+              </li>
+            <% end %>
+            <%= tag.div class: "tab-content", id: "#{dom_id(execution, :tab)}Content" do %>
+              <%= tag.div class: "tab-pane fade show active", id: dom_id(execution, :application_pane), role: "tabpane", aria: { labelledby: dom_id(execution, :application) }, tabindex: 0 do %>
+                <div class="small">
+                  <code class="text-wrap text-break m-0 text-secondary-emphasis">
+                    <%= safe_join(execution.filtered_error_backtrace, tag.br) %>
+                  </code>
+                </div>
+              <% end %>
+              <%= tag.div class: "tab-pane fade", id: dom_id(execution, :full_pane), role: "tabpane", aria: { labelledby: dom_id(execution, :full) }, tabindex: 0 do %>
+                <div class="small">
+                  <code class="text-wrap text-break m-0 text-secondary-emphasis">
+                    <%= safe_join(execution.error_backtrace, tag.br) %>
+                  </code>
+                </div>
+              <% end %>
+            <% end %>
+          <% end %>
         <% end %>
       <% end %>
       <%= render 'good_job/custom_execution_details', execution: execution, job: @job %>

data/app/views/good_job/jobs/_table.erb

@@ -25,7 +25,7 @@
             </button>
             <ul class="dropdown-menu" aria-labelledby="destroy-dropdown-toggle">
               <li>
-                <%= form.button type: 'submit', name: 'mass_action', value: 'destroy', class: 'btn', title: t(".actions.destroy_all"), data: { confirm: t(".actions.confirm_destroy_all"), disable: true } do %>
+                <%= form.button type: 'submit', name: 'mass_action', value: 'destroy', class: 'btn dropdown-item', title: t(".actions.destroy_all"), data: { confirm: t(".actions.confirm_destroy_all"), disable: true } do %>
                   <span class="me-1"><%= render_icon "trash" %></span> <%=t "good_job.actions.destroy" %>
                 <% end %>
               </li>
@@ -68,7 +68,7 @@
                 <%= tag.h5 tag.code(link_to(job.display_name, job_path(job), class: "text-reset text-decoration-none")), class: "text-reset mb-0" %>
                 <% if job.error %>
                   <div class="mt-1 small">
-                    <strong class="small"><%=t "good_job.shared.error" %>:</strong>
+                    <strong class="small text-danger"><%=t "good_job.shared.error" %>:</strong>
                     <code class="text-wrap text-break m-0 text-secondary-emphasis"><%= job.error %></code>
                   </div>
                 <% end %>

data/config/locales/de.yml

@@ -124,6 +124,8 @@ de:
       discard:
         notice: Auftrag wurde verworfen
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: in der Warteschlange
         runtime: Laufzeit
         title: Hinrichtungen

data/config/locales/en.yml

@@ -124,6 +124,8 @@ en:
       discard:
         notice: Job has been discarded
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: in queue
         runtime: runtime
         title: Executions

data/config/locales/es.yml

@@ -124,6 +124,8 @@ es:
       discard:
         notice: La tarea ha sido descartada
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: en cola
         runtime: en ejecución
         title: Ejecuciones

data/config/locales/fr.yml

@@ -124,6 +124,8 @@ fr:
       discard:
         notice: Le job a été mis au rebut
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: Dans la file d'attente
         runtime: Durée
         title: Exécutions

data/config/locales/it.yml

@@ -124,6 +124,8 @@ it:
       discard:
         notice: Il job è stato scartato
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: in coda
         runtime: tempo di esecuzione
         title: Esecuzioni

data/config/locales/ja.yml

@@ -124,6 +124,8 @@ ja:
       discard:
         notice: ジョブが破棄されました
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: 待機中
         runtime: 実行時間
         title: 実行

data/config/locales/ko.yml

@@ -124,6 +124,8 @@ ko:
       discard:
         notice: 작업이 폐기되었습니다.
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: 대기 중
         runtime: 실행 시간
         title: 실행

data/config/locales/nl.yml

@@ -33,7 +33,7 @@ nl:
         no_batches_found: Geen batches gevonden.
     cron_entries:
       actions:
-        confirm_disable: Weet u zeker dat u deze cron-vermelding wilt uitschakelen?
+        confirm_disable: Weet u zekerf dat u deze cron-vermelding wilt uitschakelen?
         confirm_enable: Weet u zeker dat u deze cron-invoer wilt inschakelen?
         confirm_enqueue: Weet u zeker dat u deze cron-vermelding in de wachtrij wilt plaatsen?
         disable: Schakel cron-invoer uit
@@ -124,6 +124,8 @@ nl:
       discard:
         notice: Job is weggegooid
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: in de wachtrij
         runtime: looptijd
         title: Executies

data/config/locales/pt-BR.yml

@@ -124,6 +124,8 @@ pt-BR:
       discard:
         notice: A tarefa foi descartada.
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: na fila
         runtime: tempo de execução
         title: Execuções

data/config/locales/ru.yml

@@ -148,6 +148,8 @@ ru:
       discard:
         notice: Задание было отменено
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: в очереди
         runtime: время выполнения
         title: Выполнение заданий

data/config/locales/tr.yml

@@ -124,6 +124,8 @@ tr:
       discard:
         notice: İş İptal Edildi
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: sırada
         runtime: çalışma süresi
         title: İşlemler

data/config/locales/uk.yml

@@ -148,6 +148,8 @@ uk:
       discard:
         notice: Завдання було відхилено
       executions:
+        application_trace: Application Trace
+        full_trace: Full Trace
         in_queue: у черзі
         runtime: час виконання
         title: Виконання

data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb

@@ -57,6 +57,7 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
       t.datetime :finished_at
       t.text :error
       t.integer :error_event, limit: 2
+      t.text :error_backtrace, array: true
     end
 
     create_table :good_job_processes, id: :uuid do |t|

data/lib/generators/good_job/templates/update/migrations/10_create_good_job_execution_error_backtrace.rb.erb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+class CreateGoodJobExecutionErrorBacktrace < ActiveRecord::Migration<%= migration_version %>
+  def change
+    reversible do |dir|
+      dir.up do
+        # Ensure this incremental update migration is idempotent
+        # with monolithic install migration.
+        return if connection.column_exists?(:good_job_executions, :error_backtrace)
+      end
+    end
+
+    add_column :good_job_executions, :error_backtrace, :text, array: true
+  end
+end

data/lib/good_job/active_job_extensions/concurrency.rb

@@ -96,7 +96,7 @@ module GoodJob
 
               query = DiscreteExecution.joins(:job)
                                        .where(GoodJob::Job.table_name => { concurrency_key: key })
-                                       .where(DiscreteExecution.arel_table[:created_at].gt(throttle_period.ago))
+                                       .where(DiscreteExecution.arel_table[:created_at].gt(DiscreteExecution.bind_value('created_at', throttle_period.ago, ActiveRecord::Type::DateTime)))
               allowed_active_job_ids = query.where(error: nil).or(query.where.not(error: "GoodJob::ActiveJobExtensions::Concurrency::ThrottleExceededError: GoodJob::ActiveJobExtensions::Concurrency::ThrottleExceededError"))
                                             .order(created_at: :asc)
                                             .limit(throttle_limit)

data/lib/good_job/active_job_extensions/notify_options.rb

@@ -9,7 +9,7 @@ module GoodJob
     # @example
     #   # Include the concern to your job class:
     #   class MyJob < ApplicationJob
-    #     include GoodJob::ActiveJobExtensions::Notify
+    #     include GoodJob::ActiveJobExtensions::NotifyOptions
     #     self.good_job_notify = false
     #   end
     #

data/lib/good_job/capsule.rb

@@ -12,7 +12,7 @@ module GoodJob
     cattr_reader :instances, default: Concurrent::Array.new, instance_reader: false
 
     # @param configuration [GoodJob::Configuration] Configuration to use for this capsule.
-    def initialize(configuration: GoodJob.configuration)
+    def initialize(configuration: nil)
       @configuration = configuration
       @startable = true
       @started_at = nil
@@ -30,13 +30,13 @@ module GoodJob
         return unless startable?(force: force)
 
         @shared_executor = GoodJob::SharedExecutor.new
-        @notifier = GoodJob::Notifier.new(enable_listening: @configuration.enable_listen_notify, executor: @shared_executor.executor)
-        @poller = GoodJob::Poller.new(poll_interval: @configuration.poll_interval)
-        @multi_scheduler = GoodJob::MultiScheduler.from_configuration(@configuration, warm_cache_on_initialize: true)
+        @notifier = GoodJob::Notifier.new(enable_listening: configuration.enable_listen_notify, executor: @shared_executor.executor)
+        @poller = GoodJob::Poller.new(poll_interval: configuration.poll_interval)
+        @multi_scheduler = GoodJob::MultiScheduler.from_configuration(configuration, warm_cache_on_initialize: true)
         @notifier.recipients.push([@multi_scheduler, :create_thread])
         @poller.recipients.push(-> { @multi_scheduler.create_thread({ fanout: true }) })
 
-        @cron_manager = GoodJob::CronManager.new(@configuration.cron_entries, start_on_initialize: true, executor: @shared_executor.executor) if @configuration.enable_cron?
+        @cron_manager = GoodJob::CronManager.new(configuration.cron_entries, start_on_initialize: true, executor: @shared_executor.executor) if configuration.enable_cron?
 
         @startable = false
         @started_at = Time.current
@@ -51,7 +51,7 @@ module GoodJob
     #   * +nil+ will trigger a shutdown but not wait for it to complete.
     # @return [void]
     def shutdown(timeout: NONE)
-      timeout = @configuration.shutdown_timeout if timeout == NONE
+      timeout = configuration.shutdown_timeout if timeout == NONE
       GoodJob._shutdown_all([@shared_executor, @notifier, @poller, @multi_scheduler, @cron_manager].compact, timeout: timeout)
       @startable = false
       @started_at = nil
@@ -101,6 +101,10 @@ module GoodJob
 
     private
 
+    def configuration
+      @configuration || GoodJob.configuration
+    end
+
     def startable?(force: false)
       !@started_at && (@startable || force)
     end

data/lib/good_job/version.rb

@@ -2,7 +2,7 @@
 
 module GoodJob
   # GoodJob gem version.
-  VERSION = '3.27.4'
+  VERSION = '3.28.0'
 
   # GoodJob version as Gem::Version object
   GEM_VERSION = Gem::Version.new(VERSION)

data/lib/good_job.rb

@@ -203,7 +203,7 @@ module GoodJob
       deleted_batches_count = 0
       deleted_discrete_executions_count = 0
 
-      jobs_query = GoodJob::Job.where('finished_at <= ?', timestamp).order(finished_at: :asc).limit(in_batches_of)
+      jobs_query = GoodJob::Job.finished_before(timestamp).order(finished_at: :asc).limit(in_batches_of)
       jobs_query = jobs_query.succeeded unless include_discarded
       loop do
         active_job_ids = jobs_query.pluck(:active_job_id)
@@ -219,7 +219,7 @@ module GoodJob
       end
 
       if GoodJob::BatchRecord.migrated?
-        batches_query = GoodJob::BatchRecord.where('finished_at <= ?', timestamp).limit(in_batches_of)
+        batches_query = GoodJob::BatchRecord.finished_before(timestamp).limit(in_batches_of)
         batches_query = batches_query.succeeded unless include_discarded
         loop do
           deleted = batches_query.delete_all

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 3.27.4
+  version: 3.28.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-04-04 00:00:00.000000000 Z
+date: 2024-04-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -94,20 +94,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.14.1
-- !ruby/object:Gem::Dependency
-  name: benchmark-ips
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: capybara
   requirement: !ruby/object:Gem::Requirement
@@ -150,20 +136,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: pry-rails
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: puma
   requirement: !ruby/object:Gem::Requirement
@@ -385,6 +357,7 @@ files:
 - lib/generators/good_job/templates/update/migrations/07_recreate_good_job_cron_indexes_with_conditional.rb.erb
 - lib/generators/good_job/templates/update/migrations/08_create_good_job_labels.rb.erb
 - lib/generators/good_job/templates/update/migrations/09_create_good_job_labels_index.rb.erb
+- lib/generators/good_job/templates/update/migrations/10_create_good_job_execution_error_backtrace.rb.erb
 - lib/generators/good_job/templates/update/migrations/10_remove_good_job_active_id_index.rb.erb
 - lib/generators/good_job/templates/update/migrations/11_create_index_good_job_jobs_for_candidate_lookup.rb.erb
 - lib/generators/good_job/update_generator.rb