delayed 4.0.1 → 4.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a2b90a867a985a3bf32841f0cadac600d9d9f076d30bb04c85845c65b0e97ba9
-  data.tar.gz: 7b527829bd45c8f4bff4b6c62ea15ecc62aeca4461e74e2ff0b6f0c5a1800243
+  metadata.gz: 7af8a042ab9fe6f3e5bb5709860ab1c7544f5f13124b02cb5d181b9afa378b6d
+  data.tar.gz: c76388de277c68702a51aa8379b83902e9f339ec0444b41e67b663a7d6db76b7
 SHA512:
-  metadata.gz: bbce505c6b79c92df2c69e56e288af6437304b8de87227058e03f31fbb9c20b6f86869c48936b170806e96469e744813e9674178d965ebdb8a4a69542da496ed
-  data.tar.gz: e6a9976ea79803f45875cddbe96639a9d2d760374ce82b8ddd3d61b7bc594abe9fcae8f2610c4e4952c026dfaf7e613db7fabed19c3bb499d3f98c0bbc6c4a93
+  metadata.gz: bd7129db0460a29f3000c630a47b3801b4b57e42de65ac12b3c719e670f48a1d955b7fc714f56dc3b6d269db819393ca562256a1305a25b28bcfa81775db97d8
+  data.tar.gz: 7b81ffcbf2eca36479c7648c8c94a878d5c3c4a2f674a94ffe08f2f6fadf570a2f0c21f997688d24e342d0296a58a9a465fdc9cac6db9c572651dfde3b6b6479

data/README.md

@@ -416,14 +416,16 @@ QUEUE=tracking rake delayed:monitor
 QUEUES=mailers,tasks rake delayed:monitor
 ```
 
-The following events will be emitted, grouped by priority name (e.g. "interactive") and queue name,
-and the metric's "`:value`" will be available in the event's payload.  **This means that there will
-be one value _per_ unique combination of queue & priority**, and totals must be computed via
-downstream aggregation (e.g. as a StatsD "gauge" metric).
+The following events will be emitted, grouped by priority name (e.g. "interactive"), queue name,
+and the values of any columns configured via [tag_columns](#tagging-metrics-with-additional-columns) (none, by default). The metric's
+"`:value`" will be available in the event's payload.  **This means that there will be one value
+_per_ unique combination of queue, priority, and tag column values**, and totals must be computed
+via downstream aggregation (e.g. as a StatsD "gauge" metric, summed or maxed by tag).
 
 - **delayed.job.count** - the total number of jobs
 - **delayed.job.future_count** - jobs where run_at is in the future
 - **delayed.job.working_count** - jobs that are currently being worked off (excludes failed jobs)
+- **delayed.job.locked_count** - jobs that are currently locked by a worker (equivalent to working_count)
 - **delayed.job.workable_count** - jobs that are waiting to be worked off
 - **delayed.job.erroring_count** - jobs where attempts > 0
 - **delayed.job.failed_count** - jobs where failed_at is not nil
@@ -432,7 +434,8 @@ downstream aggregation (e.g. as a StatsD "gauge" metric).
 
 An additional _experimental_ metric is available, intended for use with application autoscaling:
 
-- **delayed.job.alert_age_percent** - the _percent_ to which the oldest job has reached the "age alert" threshold. (See the [Alerting Threshholds](#priority-based-alerting-threshholds) section above.)
+- **delayed.job.alert_age_percent** - the _percent_ to which the oldest job has reached the "age alert"
+threshold. (See the [Alerting Threshholds](#priority-based-alerting-threshholds) section above.)
 
 All of these events may be subscribed to via a single regular expression (again, in your application
 config or in an initializer):
@@ -448,7 +451,7 @@ ActiveSupport::Notifications.subscribe(/delayed\.job\..*_(count|age|percent)/) d
 end
 ```
 
-Additionally, the monitor process with emit a **delayed.monitor.run** event with a duration
+Additionally, the monitor process will emit a **delayed.monitor.run** event with a duration
 attached, so that you can monitor the time it takes to emit these aggregate metrics.
 
 ```ruby
@@ -458,6 +461,40 @@ ActiveSupport::Notifications.subscribe('delayed.monitor.run') do |*args|
 end
 ```
 
+#### Tagging metrics with additional columns
+
+By default, the monitor only groups events by `priority` and `queue`. To add additional columns
+to the query's `GROUP BY` clause, declare them in an initializer config:
+
+```ruby
+Delayed::Monitor.tag_columns = %i(name owner)
+```
+
+Tagged series are only emitted while matching jobs exist, and an untagged zero is always emitted
+per (priority, queue) as a baseline. With `Delayed::Monitor.tag_columns = %i(name)` and one
+enqueued job, `delayed.job.count` emits:
+
+```ruby
+{ priority: 'interactive', queue: 'default', name: 'SimpleJob', value: 1 }
+{ priority: 'interactive', queue: 'default', value: 0 }
+{ priority: 'user_visible', queue: 'default', value: 0 }
+{ priority: 'eventual', queue: 'default', value: 0 }
+{ priority: 'reporting', queue: 'default', value: 0 }
+```
+
+Tag columns must already exist on the jobs table — the monitor validates this at startup and
+raises an `ArgumentError` if any are missing, so migrate a new column before adding it here.
+
+`NULL` values are emitted as `nil` tags. Your notification subscriber can decide how to represent
+these. Expect `nil`s for jobs enqueued before a newly added column was populated.
+
+**Avoid** choosing high-cardinality columns like `id` as this will result in very poor query
+performance (and may essentially turn every row into its own metric result!). Prefer low-cardinality
+columns like `name` (the name of the job class) that are worth the query performance trade-off.
+
+**You are strongly encouraged to add new indexes** incorporating those columns. When adding `tag_columns`,
+`idx_delayed_jobs_live` and `idx_delayed_jobs_failed` will no longer fully cover the monitor's queries.
+
 ## Configuration
 
 `Delayed` is highly configurable, but ships with opinionated defaults.  If you need to change any
@@ -595,6 +632,18 @@ upserting `RETURNING` clause and database-native timestamp arithmetic. You can c
 connection at runtime with `Delayed::Limit.supported?`. (Attempting to use the limiter on an
 unsupported database will raise `Delayed::Limit::UnsupportedDatabaseError`.)
 
+Additionally, the limiter **must not be called inside an open database
+transaction** (it raises `Delayed::Limit::TransactionOpenError`), because the
+reserved row would stay locked until the transaction commits. If it must run
+inside transactions, give `Delayed::Limit` a connection pool of its own (at the
+cost of up to one extra connection per thread) in a Rails initializer:
+
+```ruby
+Rails.application.config.to_prepare do
+  Delayed::Limit.establish_connection(ActiveRecord::Base.connection_db_config.configuration_hash)
+end
+```
+
 #### Traffic Shaping vs Traffic Enforcement
 
 By default, the limiter will `sleep` up to 5 seconds (or a specified `wait_timeout`) before

data/app/models/delayed/limit.rb

@@ -38,6 +38,10 @@ module Delayed
   #
   # It is **not** recommended to register limits dynamically at runtime, because
   # registered limits are cached indefinitely in memory and are not thread-safe.
+  #
+  # The limiter must not be called inside an open database transaction (it
+  # raises `TransactionOpenError`), because the reserved row would stay locked
+  # until that transaction commits.
   class Limit < ActiveRecord::Base
     self.table_name = 'delayed_limits'
 
@@ -52,6 +56,12 @@ module Delayed
     # (see `.supported?`).
     class UnsupportedDatabaseError < StandardError; end
 
+    # Raised when `.within_limit` is called inside an open database
+    # transaction. The reservation query takes a row lock that would otherwise
+    # be held until the transaction commits, causing lock contention to pile up
+    # behind the caller's (potentially long-running) work.
+    class TransactionOpenError < StandardError; end
+
     class << self
       # Used only for limits registered in advance (via `.register!`):
       def limits
@@ -115,6 +125,11 @@ module Delayed
         # e.g. for a target of 100 req/min, the drain_interval would be 0.6 sec/req.
         drain_interval = config.fetch(:per).seconds / config.fetch(:max).to_d
 
+        if transaction_open?
+          raise TransactionOpenError,
+            'Delayed::Limit.within_limit cannot be called inside a database transaction '
+        end
+
         # Attempt to reserve capacity via an uncached database query:
         limit = connection.uncached do
           find_by_sql(reserve_sql(purpose, drain_interval, wait_timeout.seconds)).first
@@ -144,6 +159,14 @@ module Delayed
 
       private
 
+      # Checks `joinable?` rather than the connection's `transaction_open?` so
+      # that the per-example transaction opened by Rails' transactional tests
+      # (which is non-joinable) does not trip the guard, while any transaction
+      # opened by application code (joinable by default) does.
+      def transaction_open?
+        connection.current_transaction.joinable?
+      end
+
       # We reserve capacity by pushing a 'drained_at' timestamp forward by the
       # drain_interval, returning how long the caller must wait to avoid filling
       # the bucket beyond its capacity. (The WHERE clause significantly reduces

data/lib/delayed/monitor.rb

@@ -17,7 +17,17 @@ module Delayed
 
     cattr_accessor :sleep_delay, instance_writer: false, default: 60
 
+    def self.tag_columns
+      @tag_columns ||= [].freeze
+    end
+
+    def self.tag_columns=(columns)
+      @tag_columns = columns.map(&:to_sym).freeze
+    end
+
     def initialize
+      validate_tag_columns!
+      @tag_columns = self.class.tag_columns
       @jobs = Job.group(:priority, :queue)
       @jobs = @jobs.where(queue: Worker.queues) if Worker.queues.any?
       @memo = {}
@@ -64,13 +74,23 @@ module Delayed
 
     private
 
-    attr_reader :jobs
+    attr_reader :jobs, :tag_columns
+
+    def validate_tag_columns!
+      if self.class.tag_columns.any? { |column| Job.column_names.exclude?(column.to_s) }
+        raise ArgumentError, "Delayed::Monitor.tag_columns includes columns missing from #{Job.table_name}. " \
+                             "Available columns: #{Job.column_names.join(', ')}"
+      end
+    end
 
     def emit_metric!(metric)
-      query_for(metric).reverse_merge(default_results).each do |(priority, queue), value|
+      query_for(metric)
+        .merge!(default_results) { |_key, existing, _default| existing }
+        .each do |(priority, queue, *column_values), value|
+        tags = column_values.zip(tag_columns).to_h { |val, column| [column, val] }
         ActiveSupport::Notifications.instrument(
           "delayed.job.#{metric}",
-          default_tags.merge(priority: Priority.new(priority).to_s, queue: queue, value: value),
+          default_tags.merge(priority: Priority.new(priority).to_s, queue: queue, **tags, value: value),
         )
       end
     end
@@ -96,24 +116,30 @@ module Delayed
     end
 
     # This method generates a query that scans the specified scope, groups by
-    # priority and queue, and calculates the specified aggregates. An outer
-    # query is executed for priority bucketing and appending db_now_utc (to
-    # avoid running these computations for each tuple in the inner query).
+    # priority and queue (plus any tag_columns), and calculates the
+    # specified aggregates. An outer query is executed for priority bucketing
+    # and appending db_now_utc (to avoid running these computations for each
+    # tuple in the inner query).
     def grouped_query(scope, include_db_time: false, **kwargs)
       inner_selects = kwargs.map { |key, (agg, expr)| as_expression(agg, expr, key) }
       outer_selects = kwargs.map { |key, (agg, _)| as_expression(agg == :count ? :sum : agg, key, key) }
       outer_selects << "#{self.class.sql_now_in_utc} AS db_now_utc" if include_db_time
 
       Delayed::Job
-        .from(scope.select(:priority, :queue, *inner_selects).group(:priority, :queue))
-        .group(priority_case_statement, :queue).select(
+        .from(scope.select(:priority, :queue, *tag_columns, *inner_selects).group(:priority, :queue, *tag_columns))
+        .group(priority_case_statement, :queue, *tag_columns).select(
           *outer_selects,
           "#{priority_case_statement} AS priority",
           'queue AS queue',
-        ).group_by { |j| [j.priority.to_i, j.queue] }
+          *tag_columns,
+        ).group_by { |j| result_key(j) }
         .transform_values(&:first)
     end
 
+    def result_key(record)
+      [record.priority.to_i, record.queue, *tag_columns.map { |column| record[column] }]
+    end
+
     def as_expression(aggregate_function, aggregate_expression, column_name)
       "#{aggregate_function.to_s.upcase}(#{aggregate_expression}) AS #{column_name}"
     end
@@ -151,10 +177,10 @@ module Delayed
     end
 
     def alert_age_percent_grouped
-      pending_counts.each_with_object({}) do |((priority, queue), j), metrics|
+      pending_counts.each_with_object({}) do |(key, j), metrics|
         max_age = time_ago(db_now(j), j.run_at)
-        alert_age = Priority.new(priority).alert_age
-        metrics[[priority, queue]] = [max_age / alert_age * 100, 100].min if alert_age
+        alert_age = Priority.new(key.first).alert_age
+        metrics[key] = [max_age / alert_age * 100, 100].min if alert_age
       end
     end
 
@@ -193,7 +219,8 @@ module Delayed
     end
 
     def failed_counts
-      @memo[:failed_counts] ||= grouped_query(jobs.failed, count: [:count, '*'])
+      @memo[:failed_counts] ||=
+        grouped_query(jobs.failed, count: [:count, '*'])
     end
 
     def db_now(record)

data/lib/delayed/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Delayed
-  VERSION = '4.0.1'
+  VERSION = '4.2.0'
 end

data/spec/delayed/__snapshots__/job_spec.rb.snap

@@ -53,8 +53,8 @@ SNAP
 snapshots["generates a postgresql query plan for multiple queues 1"] = <<-SNAP
 Index Scan using idx_delayed_jobs_live on public.delayed_jobs  (cost=...)
   Output: id, priority, attempts, handler, last_error, run_at, locked_at, failed_at, locked_by, queue, created_at, updated_at, name
-  Index Cond: (delayed_jobs.run_at <= '2025-11-10 17:20:13'::timestamp without time zone)
-  Filter: (((delayed_jobs.queue)::text = ANY ('{default,mailers,tracking}'::text[])) AND ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone) OR (((delayed_jobs.locked_by)::text = 'worker1'::text) AND (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone))))
+  Index Cond: ((delayed_jobs.run_at <= '2025-11-10 17:20:13'::timestamp without time zone) AND ((delayed_jobs.queue)::text = ANY ('{default,mailers,tracking}'::text[])))
+  Filter: ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone) OR (((delayed_jobs.locked_by)::text = 'worker1'::text) AND (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone)))
 SNAP
 
 snapshots["[legacy index] generates the expected postgresql query plan 1"] = <<-SNAP

data/spec/delayed/limit_spec.rb

@@ -3,6 +3,8 @@
 require 'helper'
 
 RSpec::Matchers.define_negated_matcher :not_change, :change
+RSpec::Matchers.define_negated_matcher :not_yield_control, :yield_control
+RSpec::Matchers.define_negated_matcher :not_emit_notification, :emit_notification
 
 RSpec.describe Delayed::Limit do
   before do
@@ -128,6 +130,105 @@ RSpec.describe Delayed::Limit do
       end
     end
 
+    context 'when called inside a database transaction' do
+      it 'raises TransactionOpenError without reserving capacity or yielding' do
+        expect { |b| described_class.transaction { described_class.within_limit(purpose, **limit, &b) } }
+          .to raise_error(described_class::TransactionOpenError, /inside a database transaction/)
+          .and not_change { described_class.count }
+          .and not_yield_control
+      end
+
+      it 'raises before the transaction is materialized (no SQL is sent)' do
+        expect { described_class.transaction { described_class.within_limit(purpose, **limit) { nil } } }
+          .to raise_error(described_class::TransactionOpenError)
+          .and not_emit_notification('sql.active_record')
+      end
+    end
+
+    context 'when the only open transaction is non-joinable' do
+      it 'does not raise, and reserves capacity as usual' do
+        expect { |b| described_class.transaction(joinable: false) { described_class.within_limit(purpose, **limit, &b) } }
+          .to yield_control.once
+
+        expect(described_class.count).to eq(1)
+      end
+
+      it 'still raises when application code opens a nested transaction' do
+        expect { |b| described_class.transaction(joinable: false) { described_class.transaction { described_class.within_limit(purpose, **limit, &b) } } }
+          .to raise_error(described_class::TransactionOpenError)
+          .and not_yield_control
+      end
+    end
+
+    context "under Rails' transactional test harness" do
+      # Opens the per-example transaction the same way ActiveRecord::TestFixtures does
+      # (via the connection pool on Rails 7.2+, and directly on the connection before that).
+      around do |example|
+        pool = described_class.connection_pool
+        if pool.respond_to?(:pin_connection!)
+          pool.pin_connection!(true)
+          begin
+            example.run
+          ensure
+            pool.unpin_connection!
+          end
+        else
+          connection = described_class.connection
+          connection.begin_transaction(joinable: false, _lazy: false)
+          begin
+            example.run
+          ensure
+            connection.rollback_transaction
+          end
+        end
+      end
+
+      it 'does not raise, and reserves capacity as usual' do
+        expect(described_class.connection.transaction_open?).to be(true)
+
+        expect { |b| described_class.within_limit(purpose, **limit, &b) }
+          .to yield_control.once
+
+        expect(described_class.count).to eq(1)
+      end
+
+      it 'still raises when the code under test opens a transaction' do
+        expect { |b| described_class.transaction { described_class.within_limit(purpose, **limit, &b) } }
+          .to raise_error(described_class::TransactionOpenError)
+          .and not_yield_control
+      end
+    end
+
+    context 'when Delayed::Limit has its own connection pool' do
+      around do |example|
+        config = if ActiveRecord::Base.respond_to?(:connection_db_config)
+          ActiveRecord::Base.connection_db_config.configuration_hash
+        else
+          ActiveRecord::Base.connection_config
+        end
+        described_class.establish_connection(config)
+        example.run
+      ensure
+        described_class.remove_connection
+      end
+
+      it 'uses a separate pool from the application' do
+        expect(described_class.connection_pool).not_to equal(ActiveRecord::Base.connection_pool)
+      end
+
+      it 'does not treat a transaction on the application pool as an open transaction' do
+        expect { |b| ActiveRecord::Base.transaction { described_class.within_limit(purpose, **limit, &b) } }
+          .to yield_control.once
+          .and change { described_class.count }.by(1)
+      end
+
+      it 'still raises for a transaction on its own pool' do
+        expect { |b| described_class.transaction { described_class.within_limit(purpose, **limit, &b) } }
+          .to raise_error(described_class::TransactionOpenError)
+          .and not_yield_control
+      end
+    end
+
     context 'with a limit registered in advance via .register!' do
       before { allow(described_class).to receive(:limits).and_return(purpose => limit) }
 

data/spec/delayed/monitor_spec.rb

@@ -14,6 +14,12 @@ RSpec.describe Delayed::Monitor do
     }
   end
 
+  describe '.tag_columns' do
+    it 'defaults to no tag columns' do
+      expect(described_class.tag_columns).to eq []
+    end
+  end
+
   describe '#run!' do
     let(:app_local_db_time) { false }
 
@@ -201,6 +207,91 @@ RSpec.describe Delayed::Monitor do
           .and emit_notification("delayed.job.max_age").with_payload(p30_payload.merge(queue: 'banana')).approximately.with_value(4.hours)
       end
 
+      context 'when tag_columns is set to name' do
+        around do |example|
+          described_class.tag_columns = %i(name)
+          example.run
+        ensure
+          described_class.tag_columns = []
+        end
+
+        let(:named_payload) { p0_payload.merge(name: 'SimpleJob') }
+
+        it 'tags each series with the job name' do
+          expect { subject.run! }
+            .to emit_notification("delayed.job.max_age").with_payload(named_payload).approximately.with_value(30.seconds)
+            .and emit_notification("delayed.job.failed_count").with_payload(named_payload).with_value(1)
+        end
+
+        context 'when multiple job names share a priority and queue' do
+          let!(:other_named_job) { Delayed::Job.create! p0_attributes.merge(name: 'OtherJob', run_at: now - 10.minutes) }
+
+          it 'emits a separate series per name' do
+            expect { subject.run! }
+              .to emit_notification("delayed.job.max_age").with_payload(named_payload).approximately.with_value(30.seconds)
+              .and emit_notification("delayed.job.max_age").with_payload(named_payload.merge(name: 'OtherJob')).approximately.with_value(10.minutes)
+          end
+        end
+
+        context 'when a job predates the name column' do
+          around do |example|
+            ValidateRunAtAndNameNotNull.migrate(:down)
+            AddRunAtAndNameNotNullCheck.migrate(:down)
+            example.run
+          ensure
+            Delayed::Job.delete_all
+            AddRunAtAndNameNotNullCheck.migrate(:up)
+            ValidateRunAtAndNameNotNull.migrate(:up)
+          end
+
+          let!(:unnamed_job) { Delayed::Job.create! p0_attributes.merge(name: nil, run_at: now - 10.minutes) }
+
+          it 'emits metrics with a nil name' do
+            expect { subject.run! }
+              .to emit_notification("delayed.job.max_age").with_payload(named_payload.merge(name: nil)).approximately.with_value(10.minutes)
+          end
+        end
+      end
+
+      context 'when tag_columns includes a custom column' do
+        around do |example|
+          Delayed::Job.connection.add_column :delayed_jobs, :owner, :string
+          Delayed::Job.reset_column_information
+          described_class.tag_columns = %i(name owner)
+          example.run
+        ensure
+          described_class.tag_columns = []
+          Delayed::Job.connection.remove_column :delayed_jobs, :owner
+          Delayed::Job.reset_column_information
+        end
+
+        let(:named_payload) { p0_payload.merge(name: 'SimpleJob') }
+        let!(:team_a_job) { Delayed::Job.create! p0_attributes.merge(owner: 'team_a', run_at: now - 10.minutes) }
+        let!(:team_b_job) { Delayed::Job.create! p0_attributes.merge(owner: 'team_b', run_at: now - 20.minutes) }
+
+        it "tags each series with the column's value, passing NULLs through as nil" do
+          expect { subject.run! }
+            .to emit_notification("delayed.job.max_age").with_payload(named_payload.merge(owner: 'team_a')).approximately.with_value(10.minutes)
+            .and emit_notification("delayed.job.max_age").with_payload(named_payload.merge(owner: 'team_b')).approximately.with_value(20.minutes)
+            .and emit_notification("delayed.job.max_age").with_payload(named_payload.merge(owner: nil)).approximately.with_value(30.seconds)
+            .and emit_notification("delayed.job.failed_count").with_payload(named_payload.merge(owner: nil)).with_value(1)
+        end
+      end
+
+      context 'when tag_columns names a column that does not exist' do
+        around do |example|
+          described_class.tag_columns = %i(name missing_column)
+          example.run
+        ensure
+          described_class.tag_columns = []
+        end
+
+        it 'raises loudly at monitor startup rather than skipping the column' do
+          expect { described_class.new }
+            .to raise_error(ArgumentError, /tag_columns includes columns missing from delayed_jobs\. Available columns: .*\bname\b/)
+        end
+      end
+
       context 'when named priorities are customized' do
         around do |example|
           Delayed::Priority.names = { high: 0, low: 20 }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: delayed
 version: !ruby/object:Gem::Version
-  version: 4.0.1
+  version: 4.2.0
 platform: ruby
 authors:
 - Nathan Griffith