delayed 4.0.1 → 4.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a2b90a867a985a3bf32841f0cadac600d9d9f076d30bb04c85845c65b0e97ba9
-  data.tar.gz: 7b527829bd45c8f4bff4b6c62ea15ecc62aeca4461e74e2ff0b6f0c5a1800243
+  metadata.gz: 70e3822f06e3d5b6a663f08998abf783ae91137a96cf154dded1487548ab9544
+  data.tar.gz: e166f6ba77ff5e268059dc32de8e72e619b56106e69a96b884c1c5e31e22542a
 SHA512:
-  metadata.gz: bbce505c6b79c92df2c69e56e288af6437304b8de87227058e03f31fbb9c20b6f86869c48936b170806e96469e744813e9674178d965ebdb8a4a69542da496ed
-  data.tar.gz: e6a9976ea79803f45875cddbe96639a9d2d760374ce82b8ddd3d61b7bc594abe9fcae8f2610c4e4952c026dfaf7e613db7fabed19c3bb499d3f98c0bbc6c4a93
+  metadata.gz: 86379222dec057f27b106602cc91ce90800861f705e16fb53857590f6b730a6e523780be0eeec09c6a9dd8d9f9b1760b0c15793c73fcfac5173bf2ba17354f4b
+  data.tar.gz: 4ce59e9dbec2d10339cf3979ff5c49a97adc337fba6119d69662e65ff2407f8fc1ab32cac09ab3ad83044e795384f2c36ae478fa5763a3578b87178cdb624123

data/README.md

@@ -416,14 +416,16 @@ QUEUE=tracking rake delayed:monitor
 QUEUES=mailers,tasks rake delayed:monitor
 ```
 
-The following events will be emitted, grouped by priority name (e.g. "interactive") and queue name,
-and the metric's "`:value`" will be available in the event's payload.  **This means that there will
-be one value _per_ unique combination of queue & priority**, and totals must be computed via
-downstream aggregation (e.g. as a StatsD "gauge" metric).
+The following events will be emitted, grouped by priority name (e.g. "interactive"), queue name,
+and the values of any columns configured via [tag_columns](#tagging-metrics-with-additional-columns) (none, by default). The metric's
+"`:value`" will be available in the event's payload.  **This means that there will be one value
+_per_ unique combination of queue, priority, and tag column values**, and totals must be computed
+via downstream aggregation (e.g. as a StatsD "gauge" metric, summed or maxed by tag).
 
 - **delayed.job.count** - the total number of jobs
 - **delayed.job.future_count** - jobs where run_at is in the future
 - **delayed.job.working_count** - jobs that are currently being worked off (excludes failed jobs)
+- **delayed.job.locked_count** - jobs that are currently locked by a worker (equivalent to working_count)
 - **delayed.job.workable_count** - jobs that are waiting to be worked off
 - **delayed.job.erroring_count** - jobs where attempts > 0
 - **delayed.job.failed_count** - jobs where failed_at is not nil
@@ -432,7 +434,8 @@ downstream aggregation (e.g. as a StatsD "gauge" metric).
 
 An additional _experimental_ metric is available, intended for use with application autoscaling:
 
-- **delayed.job.alert_age_percent** - the _percent_ to which the oldest job has reached the "age alert" threshold. (See the [Alerting Threshholds](#priority-based-alerting-threshholds) section above.)
+- **delayed.job.alert_age_percent** - the _percent_ to which the oldest job has reached the "age alert"
+threshold. (See the [Alerting Threshholds](#priority-based-alerting-threshholds) section above.)
 
 All of these events may be subscribed to via a single regular expression (again, in your application
 config or in an initializer):
@@ -448,7 +451,7 @@ ActiveSupport::Notifications.subscribe(/delayed\.job\..*_(count|age|percent)/) d
 end
 ```
 
-Additionally, the monitor process with emit a **delayed.monitor.run** event with a duration
+Additionally, the monitor process will emit a **delayed.monitor.run** event with a duration
 attached, so that you can monitor the time it takes to emit these aggregate metrics.
 
 ```ruby
@@ -458,6 +461,40 @@ ActiveSupport::Notifications.subscribe('delayed.monitor.run') do |*args|
 end
 ```
 
+#### Tagging metrics with additional columns
+
+By default, the monitor only groups events by `priority` and `queue`. To add additional columns
+to the query's `GROUP BY` clause, declare them in an initializer config:
+
+```ruby
+Delayed::Monitor.tag_columns = %i(name owner)
+```
+
+Tagged series are only emitted while matching jobs exist, and an untagged zero is always emitted
+per (priority, queue) as a baseline. With `Delayed::Monitor.tag_columns = %i(name)` and one
+enqueued job, `delayed.job.count` emits:
+
+```ruby
+{ priority: 'interactive', queue: 'default', name: 'SimpleJob', value: 1 }
+{ priority: 'interactive', queue: 'default', value: 0 }
+{ priority: 'user_visible', queue: 'default', value: 0 }
+{ priority: 'eventual', queue: 'default', value: 0 }
+{ priority: 'reporting', queue: 'default', value: 0 }
+```
+
+Tag columns must already exist on the jobs table — the monitor validates this at startup and
+raises an `ArgumentError` if any are missing, so migrate a new column before adding it here.
+
+`NULL` values are emitted as `nil` tags. Your notification subscriber can decide how to represent
+these. Expect `nil`s for jobs enqueued before a newly added column was populated.
+
+**Avoid** choosing high-cardinality columns like `id` as this will result in very poor query
+performance (and may essentially turn every row into its own metric result!). Prefer low-cardinality
+columns like `name` (the name of the job class) that are worth the query performance trade-off.
+
+**You are strongly encouraged to add new indexes** incorporating those columns. When adding `tag_columns`,
+`idx_delayed_jobs_live` and `idx_delayed_jobs_failed` will no longer fully cover the monitor's queries.
+
 ## Configuration
 
 `Delayed` is highly configurable, but ships with opinionated defaults.  If you need to change any

data/lib/delayed/monitor.rb

@@ -17,7 +17,17 @@ module Delayed
 
     cattr_accessor :sleep_delay, instance_writer: false, default: 60
 
+    def self.tag_columns
+      @tag_columns ||= [].freeze
+    end
+
+    def self.tag_columns=(columns)
+      @tag_columns = columns.map(&:to_sym).freeze
+    end
+
     def initialize
+      validate_tag_columns!
+      @tag_columns = self.class.tag_columns
       @jobs = Job.group(:priority, :queue)
       @jobs = @jobs.where(queue: Worker.queues) if Worker.queues.any?
       @memo = {}
@@ -64,13 +74,23 @@ module Delayed
 
     private
 
-    attr_reader :jobs
+    attr_reader :jobs, :tag_columns
+
+    def validate_tag_columns!
+      if self.class.tag_columns.any? { |column| Job.column_names.exclude?(column.to_s) }
+        raise ArgumentError, "Delayed::Monitor.tag_columns includes columns missing from #{Job.table_name}. " \
+                             "Available columns: #{Job.column_names.join(', ')}"
+      end
+    end
 
     def emit_metric!(metric)
-      query_for(metric).reverse_merge(default_results).each do |(priority, queue), value|
+      query_for(metric)
+        .merge!(default_results) { |_key, existing, _default| existing }
+        .each do |(priority, queue, *column_values), value|
+        tags = column_values.zip(tag_columns).to_h { |val, column| [column, val] }
         ActiveSupport::Notifications.instrument(
           "delayed.job.#{metric}",
-          default_tags.merge(priority: Priority.new(priority).to_s, queue: queue, value: value),
+          default_tags.merge(priority: Priority.new(priority).to_s, queue: queue, **tags, value: value),
         )
       end
     end
@@ -96,24 +116,30 @@ module Delayed
     end
 
     # This method generates a query that scans the specified scope, groups by
-    # priority and queue, and calculates the specified aggregates. An outer
-    # query is executed for priority bucketing and appending db_now_utc (to
-    # avoid running these computations for each tuple in the inner query).
+    # priority and queue (plus any tag_columns), and calculates the
+    # specified aggregates. An outer query is executed for priority bucketing
+    # and appending db_now_utc (to avoid running these computations for each
+    # tuple in the inner query).
     def grouped_query(scope, include_db_time: false, **kwargs)
       inner_selects = kwargs.map { |key, (agg, expr)| as_expression(agg, expr, key) }
       outer_selects = kwargs.map { |key, (agg, _)| as_expression(agg == :count ? :sum : agg, key, key) }
       outer_selects << "#{self.class.sql_now_in_utc} AS db_now_utc" if include_db_time
 
       Delayed::Job
-        .from(scope.select(:priority, :queue, *inner_selects).group(:priority, :queue))
-        .group(priority_case_statement, :queue).select(
+        .from(scope.select(:priority, :queue, *tag_columns, *inner_selects).group(:priority, :queue, *tag_columns))
+        .group(priority_case_statement, :queue, *tag_columns).select(
           *outer_selects,
           "#{priority_case_statement} AS priority",
           'queue AS queue',
-        ).group_by { |j| [j.priority.to_i, j.queue] }
+          *tag_columns,
+        ).group_by { |j| result_key(j) }
         .transform_values(&:first)
     end
 
+    def result_key(record)
+      [record.priority.to_i, record.queue, *tag_columns.map { |column| record[column] }]
+    end
+
     def as_expression(aggregate_function, aggregate_expression, column_name)
       "#{aggregate_function.to_s.upcase}(#{aggregate_expression}) AS #{column_name}"
     end
@@ -151,10 +177,10 @@ module Delayed
     end
 
     def alert_age_percent_grouped
-      pending_counts.each_with_object({}) do |((priority, queue), j), metrics|
+      pending_counts.each_with_object({}) do |(key, j), metrics|
         max_age = time_ago(db_now(j), j.run_at)
-        alert_age = Priority.new(priority).alert_age
-        metrics[[priority, queue]] = [max_age / alert_age * 100, 100].min if alert_age
+        alert_age = Priority.new(key.first).alert_age
+        metrics[key] = [max_age / alert_age * 100, 100].min if alert_age
       end
     end
 
@@ -193,7 +219,8 @@ module Delayed
     end
 
     def failed_counts
-      @memo[:failed_counts] ||= grouped_query(jobs.failed, count: [:count, '*'])
+      @memo[:failed_counts] ||=
+        grouped_query(jobs.failed, count: [:count, '*'])
     end
 
     def db_now(record)

data/lib/delayed/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Delayed
-  VERSION = '4.0.1'
+  VERSION = '4.1.0'
 end

data/spec/delayed/__snapshots__/job_spec.rb.snap

@@ -53,8 +53,8 @@ SNAP
 snapshots["generates a postgresql query plan for multiple queues 1"] = <<-SNAP
 Index Scan using idx_delayed_jobs_live on public.delayed_jobs  (cost=...)
   Output: id, priority, attempts, handler, last_error, run_at, locked_at, failed_at, locked_by, queue, created_at, updated_at, name
-  Index Cond: (delayed_jobs.run_at <= '2025-11-10 17:20:13'::timestamp without time zone)
-  Filter: (((delayed_jobs.queue)::text = ANY ('{default,mailers,tracking}'::text[])) AND ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone) OR (((delayed_jobs.locked_by)::text = 'worker1'::text) AND (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone))))
+  Index Cond: ((delayed_jobs.run_at <= '2025-11-10 17:20:13'::timestamp without time zone) AND ((delayed_jobs.queue)::text = ANY ('{default,mailers,tracking}'::text[])))
+  Filter: ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone) OR (((delayed_jobs.locked_by)::text = 'worker1'::text) AND (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone)))
 SNAP
 
 snapshots["[legacy index] generates the expected postgresql query plan 1"] = <<-SNAP

data/spec/delayed/monitor_spec.rb

@@ -14,6 +14,12 @@ RSpec.describe Delayed::Monitor do
     }
   end
 
+  describe '.tag_columns' do
+    it 'defaults to no tag columns' do
+      expect(described_class.tag_columns).to eq []
+    end
+  end
+
   describe '#run!' do
     let(:app_local_db_time) { false }
 
@@ -201,6 +207,91 @@ RSpec.describe Delayed::Monitor do
           .and emit_notification("delayed.job.max_age").with_payload(p30_payload.merge(queue: 'banana')).approximately.with_value(4.hours)
       end
 
+      context 'when tag_columns is set to name' do
+        around do |example|
+          described_class.tag_columns = %i(name)
+          example.run
+        ensure
+          described_class.tag_columns = []
+        end
+
+        let(:named_payload) { p0_payload.merge(name: 'SimpleJob') }
+
+        it 'tags each series with the job name' do
+          expect { subject.run! }
+            .to emit_notification("delayed.job.max_age").with_payload(named_payload).approximately.with_value(30.seconds)
+            .and emit_notification("delayed.job.failed_count").with_payload(named_payload).with_value(1)
+        end
+
+        context 'when multiple job names share a priority and queue' do
+          let!(:other_named_job) { Delayed::Job.create! p0_attributes.merge(name: 'OtherJob', run_at: now - 10.minutes) }
+
+          it 'emits a separate series per name' do
+            expect { subject.run! }
+              .to emit_notification("delayed.job.max_age").with_payload(named_payload).approximately.with_value(30.seconds)
+              .and emit_notification("delayed.job.max_age").with_payload(named_payload.merge(name: 'OtherJob')).approximately.with_value(10.minutes)
+          end
+        end
+
+        context 'when a job predates the name column' do
+          around do |example|
+            ValidateRunAtAndNameNotNull.migrate(:down)
+            AddRunAtAndNameNotNullCheck.migrate(:down)
+            example.run
+          ensure
+            Delayed::Job.delete_all
+            AddRunAtAndNameNotNullCheck.migrate(:up)
+            ValidateRunAtAndNameNotNull.migrate(:up)
+          end
+
+          let!(:unnamed_job) { Delayed::Job.create! p0_attributes.merge(name: nil, run_at: now - 10.minutes) }
+
+          it 'emits metrics with a nil name' do
+            expect { subject.run! }
+              .to emit_notification("delayed.job.max_age").with_payload(named_payload.merge(name: nil)).approximately.with_value(10.minutes)
+          end
+        end
+      end
+
+      context 'when tag_columns includes a custom column' do
+        around do |example|
+          Delayed::Job.connection.add_column :delayed_jobs, :owner, :string
+          Delayed::Job.reset_column_information
+          described_class.tag_columns = %i(name owner)
+          example.run
+        ensure
+          described_class.tag_columns = []
+          Delayed::Job.connection.remove_column :delayed_jobs, :owner
+          Delayed::Job.reset_column_information
+        end
+
+        let(:named_payload) { p0_payload.merge(name: 'SimpleJob') }
+        let!(:team_a_job) { Delayed::Job.create! p0_attributes.merge(owner: 'team_a', run_at: now - 10.minutes) }
+        let!(:team_b_job) { Delayed::Job.create! p0_attributes.merge(owner: 'team_b', run_at: now - 20.minutes) }
+
+        it "tags each series with the column's value, passing NULLs through as nil" do
+          expect { subject.run! }
+            .to emit_notification("delayed.job.max_age").with_payload(named_payload.merge(owner: 'team_a')).approximately.with_value(10.minutes)
+            .and emit_notification("delayed.job.max_age").with_payload(named_payload.merge(owner: 'team_b')).approximately.with_value(20.minutes)
+            .and emit_notification("delayed.job.max_age").with_payload(named_payload.merge(owner: nil)).approximately.with_value(30.seconds)
+            .and emit_notification("delayed.job.failed_count").with_payload(named_payload.merge(owner: nil)).with_value(1)
+        end
+      end
+
+      context 'when tag_columns names a column that does not exist' do
+        around do |example|
+          described_class.tag_columns = %i(name missing_column)
+          example.run
+        ensure
+          described_class.tag_columns = []
+        end
+
+        it 'raises loudly at monitor startup rather than skipping the column' do
+          expect { described_class.new }
+            .to raise_error(ArgumentError, /tag_columns includes columns missing from delayed_jobs\. Available columns: .*\bname\b/)
+        end
+      end
+
       context 'when named priorities are customized' do
         around do |example|
           Delayed::Priority.names = { high: 0, low: 20 }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: delayed
 version: !ruby/object:Gem::Version
-  version: 4.0.1
+  version: 4.1.0
 platform: ruby
 authors:
 - Nathan Griffith