solid_queue 1.7.0 → 1.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 15044fdadc1f5170f909ebfeeb3f0d31e948bb51edb336aa040ad477b9d12545
-  data.tar.gz: 566ab7ad221caa7c96ea1f755c8e5537ac4e37304afbb7ced6697eb1a120033b
+  metadata.gz: d55976c5a8d2a3e411cfb10099c0bbbeedc4652fcb02edfb8474b31b793c164c
+  data.tar.gz: 5a4a9ddffa329ef4ff6feb7fb642f18baaae637d8248f55bfa739d56a7f16824
 SHA512:
-  metadata.gz: 0744c1361eddc92f7cc24f26d3ca6194c7a06474cc2e3a287b3935e86b6e4dbb88c595b70ef63327431f687b8578770805a68cb33729aecb1ef7ca895b3d8e07
-  data.tar.gz: 2a40a741c8269c3a31baec31970addfc268d65f9d3bdc6130b75a1e908f22ecce2a0786b34c8dfd6c88cb92a024ca3fb84709f6d1c79e4f00022880ebd55056f
+  metadata.gz: 8301a9c5150cc8f0a99b47755fd2e22e41832a913618fffe3280bd61be27c7f84a570ba2ade570a8228b6d07b4587b914fa61b626b24e87b4d57f8cd702e9949
+  data.tar.gz: af764dc8c6749f5a88a50993693b36450a0dfe158ea04a6dd267fd23476208c858e96a17cf767152be75eb6bbc0bfb82d1cdfb3863091b5cfc4d8430eec72997

data/README.md

@@ -401,6 +401,8 @@ The supervisor is in charge of managing these processes, and it responds to the
 
 When receiving a `QUIT` signal, if workers still have jobs in-flight, these will be returned to the queue when the processes are deregistered.
 
+In `async` mode, workers, dispatchers and the scheduler are threads inside the supervisor's process, so they can't be signaled one by one. On `TERM` or `INT`, the supervisor stops them and waits up to `SolidQueue.shutdown_timeout`. If any are still running by then, it deregisters itself, which returns their in-flight jobs to the queue, and exits. On `QUIT`, it exits immediately without deregistering anything, so in-flight jobs remain claimed until another supervisor prunes these processes and marks the jobs as failed with a `SolidQueue::Processes::ProcessPrunedError`.
+
 On Windows, the `QUIT` signal can't be trapped, so the supervisor only responds to `TERM` and `INT` there.
 
 If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. Jobs that were claimed by processes with an expired heartbeat will be marked as failed with a `SolidQueue::Processes::ProcessPrunedError`. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
@@ -418,6 +420,16 @@ You can configure the database used by Solid Queue via the `config.solid_queue.c
 
 All the options available to Active Record for multiple databases can be used here.
 
+If you use MySQL or MariaDB, consider running the queue database with the `READ COMMITTED` transaction isolation level. Under the default `REPEATABLE READ`, InnoDB takes gap locks on the indexes Solid Queue polls, which under heavy load can occasionally deadlock jobs being enqueued against jobs being claimed or dispatched. `READ COMMITTED` avoids these gap locks and is perfectly safe for Solid Queue's own tables, and it's how we run it ourselves. You can set it per connection in your `database.yml`:
+
+```yaml
+queue:
+  <<: *default
+  database: my_app_queue
+  variables:
+    transaction_isolation: READ-COMMITTED
+```
+
 ### Other configuration settings
 
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
@@ -438,7 +450,7 @@ There are several settings that control how Solid Queue works that you can set a
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
 - `fork_boot_timeout`: how long a forked process can take to finish booting before the supervisor replaces it—defaults to 5 minutes. It only applies in the default `fork` mode.
 - `shutdown_timeout`: time the supervisor will wait since it sent the `TERM` signal to its supervised processes before sending a `QUIT` version to them requesting immediate termination—defaults to 5 seconds.
-- `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to `true`.
+- `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to `true`. On Rails 8.2 and later, you can go further and disable SQL notifications for the whole queue database connection by setting `sql_notifications: false` on it in your `database.yml`. This silences not only polling but also heartbeats, semaphores, maintenance queries, and everything else Solid Queue does on that connection, both in the logs and for any subscribers to `sql.active_record`.
 - `supervisor_pidfile`: path to a pidfile that the supervisor will create when booting to prevent running more than one supervisor in the same host, or in case you want to use it for a health check. It's `nil` by default.
 - `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to `true`.
 - `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true — defaults to 1 day. When installing Solid Queue, [a recurring job](#recurring-tasks) is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the `recurring.yml` configuration to change this as you see fit.
@@ -545,6 +557,8 @@ It's important to note that after one or more candidate jobs are unblocked (eith
 
 When using `discard` as the behaviour to handle conflicts, you might have jobs discarded for until the `duration` interval if something happens and a running job fails to release the semaphore.
 
+If a job's class no longer exists by the time its concurrency controls are checked—say it was renamed or removed in a deploy while jobs referencing it were still in the queue—the job is marked as failed with a `SolidQueue::Job::ClassMissingError`, so it shows up in [failed jobs](#failed-jobs-and-retries), where it can be retried once the class is back, or discarded. Jobs with a missing class picked up by a worker fail with the same error.
+
 
 For example:
 ```ruby
@@ -674,6 +688,8 @@ class ApplicationMailer < ActionMailer::Base
 
 When a process dies without a clean shutdown (for example, `SIGKILL`ed by the OS or the container runtime because of memory limits), the jobs it was running can't be released back to their queues. Once another process notices the missing heartbeats and prunes the dead process's registration, its in-flight jobs are marked as failed with `SolidQueue::Processes::ProcessPrunedError`. Solid Queue deliberately doesn't retry these automatically: the job itself might be what's killing the process (for example, a job that exhausts the container's memory), and retrying it blindly would just kill the next worker too.
 
+Note that Active Job's `retry_on` and `rescue_from` have no effect on these errors: they only intercept exceptions raised while your job's `perform` method runs, and `ProcessPrunedError` (as well as `ProcessExitError` and `ProcessMissingError`) is never raised inside the job. The process that was running the job is gone by then — the error is recorded directly as a failed execution by a *different* process, after the fact, so there's no job execution left for Active Job's retry machinery to hook into. To retry these jobs, act on the failed executions from the outside instead: manually, via [Mission Control — Jobs](https://github.com/rails/mission_control-jobs), or automatically, with a subscription like the one below.
+
 If you know your jobs are idempotent and want to implement your own recovery policy, you can subscribe to the `fail_many_claimed.solid_queue` event, which includes the error and the affected job IDs in its payload:
 
 ```ruby
@@ -812,6 +828,8 @@ that you set in production only. This is what Rails 8's default Puma config look
 
 **Note**: phased restarts are not supported currently because the plugin requires [app preloading](https://github.com/puma/puma?tab=readme-ov-file#cluster-mode) to work.
 
+The plugin ties the two processes together: if Puma goes away, the supervisor stops, and if the supervisor exits, the plugin stops Puma so that whatever manages Puma (systemd, Kamal, your container orchestrator...) restarts both. In particular, the queue database needs to be reachable when the supervisor boots, as it registers itself and cleans up after previous runs at that point; if it isn't, the supervisor exits, and Puma with it. Once running, a transient database outage doesn't bring the supervisor down: workers and dispatchers will fail and be replaced until the database is back, and then pick up where they left off.
+
 ### Running as a fork or asynchronously
 
 By default, the Puma plugin will fork additional processes for each worker and dispatcher so that they run in different processes. This provides the best isolation and performance, but can have additional memory usage.
@@ -911,6 +929,8 @@ It's possible to run multiple schedulers with the same `recurring_tasks` configu
 
 **Note**: a single recurring schedule is supported, so you can have multiple schedulers using the same schedule, but not multiple schedulers using different configurations.
 
+When a scheduler starts, it removes any static recurring tasks that are no longer in its configuration. If you remove all of them, though, no scheduler starts, so the old tasks stay in the `solid_queue_recurring_tasks` table, and in Mission Control, even though nothing runs them. You can delete them from a console with `SolidQueue::RecurringTask.static.delete_all`.
+
 Finally, it's possible to configure jobs that aren't handled by Solid Queue. That is, you can have a job like this in your app:
 ```ruby
 class MyResqueJob < ApplicationJob

data/app/models/solid_queue/batch_execution.rb

@@ -47,6 +47,9 @@ module SolidQueue
         if batch = Batch.select(:id, :finished_at, :enqueued_at).find_by(id: batch_id)
           batch.finish
         end
+      rescue => error
+        # The job is already committed; let the sweeper retry batch completion.
+        SolidQueue.instrument(:batch_progress_error, batch_id: batch_id, job_id: job_id, error: error)
       end
   end
 end

data/app/models/solid_queue/blocked_execution.rb

@@ -46,7 +46,12 @@ module SolidQueue
     def release
       SolidQueue.instrument(:release_blocked, job_id: job.id, concurrency_key: concurrency_key, released: false) do |payload|
         transaction do
-          if acquire_concurrency_lock
+          if job.job_class.nil?
+            job.failed_with(Job::ClassMissingError.for(job))
+            destroy!
+
+            payload[:failed] = true
+          elsif acquire_concurrency_lock
             promote_to_ready
             destroy!
 
@@ -58,7 +63,8 @@ module SolidQueue
 
     private
       def set_expires_at
-        self.expires_at = job.concurrency_duration.from_now
+        duration = job.job_class ? job.concurrency_duration : SolidQueue.default_concurrency_control_period
+        self.expires_at = duration.from_now
       end
 
       def acquire_concurrency_lock

data/app/models/solid_queue/claimed_execution.rb

@@ -1,126 +1,160 @@
 # frozen_string_literal: true
 
-class SolidQueue::ClaimedExecution < SolidQueue::Execution
-  belongs_to :process
+module SolidQueue
+  class ClaimedExecution < Execution
+    belongs_to :process
 
-  scope :orphaned, -> { where.missing(:process) }
+    scope :orphaned, -> { where.missing(:process) }
 
-  class Result < Struct.new(:success, :error)
-    def success?
-      success
+    class Result < Struct.new(:success, :error)
+      def success?
+        success
+      end
+    end
+
+    # Raised when a job has already run (or failed) but we couldn't update its
+    # claim/finished state because of a transient error. The claim is still held
+    # by a living worker, so it won't be recovered as orphaned unless the worker
+    # is stopped and replaced.
+    class FinalizationError < Processes::UnrecoverableError
+      def initialize(claimed_execution, cause:)
+        super("Failed to finalize claimed execution #{claimed_execution.id} (job #{claimed_execution.job_id}): #{cause.class}: #{cause.message}")
+        set_backtrace(cause.backtrace) if cause.backtrace
+      end
     end
-  end
 
-  class << self
-    def claiming(job_ids, process_id, &block)
-      job_data = Array(job_ids).collect { |job_id| { job_id: job_id, process_id: process_id } }
+    class << self
+      def claiming(job_ids, process_id, &block)
+        job_data = Array(job_ids).collect { |job_id| { job_id: job_id, process_id: process_id } }
 
-      SolidQueue.instrument(:claim, process_id: process_id, job_ids: job_ids) do |payload|
-        insert_all!(job_data)
-        where(job_id: job_ids, process_id: process_id).load.tap do |claimed|
-          block.call(claimed)
+        SolidQueue.instrument(:claim, process_id: process_id, job_ids: job_ids) do |payload|
+          insert_all!(job_data)
+          where(job_id: job_ids, process_id: process_id).load.tap do |claimed|
+            block.call(claimed)
 
-          payload[:size] = claimed.size
-          payload[:claimed_job_ids] = claimed.map(&:job_id)
+            payload[:size] = claimed.size
+            payload[:claimed_job_ids] = claimed.map(&:job_id)
+          end
         end
       end
-    end
 
-    def release_all
-      SolidQueue.instrument(:release_many_claimed) do |payload|
-        includes(:job).tap do |executions|
-          executions.each(&:release)
+      def release_all
+        SolidQueue.instrument(:release_many_claimed) do |payload|
+          includes(:job).tap do |executions|
+            executions.each(&:release)
 
-          payload[:size] = executions.size
+            payload[:size] = executions.size
+          end
         end
       end
-    end
 
-    def fail_all_with(error)
-      includes(:job).tap do |executions|
-        return if executions.empty?
+      def fail_all_with(error)
+        includes(:job).tap do |executions|
+          return if executions.empty?
+
+          SolidQueue.instrument(:fail_many_claimed) do |payload|
+            executions.each do |execution|
+              execution.failed_with(error)
+            end
 
-        SolidQueue.instrument(:fail_many_claimed) do |payload|
-          executions.each do |execution|
-            execution.failed_with(error)
+            payload[:process_ids] = executions.map(&:process_id).uniq
+            payload[:job_ids] = executions.map(&:job_id).uniq
+            payload[:size] = executions.size
+            payload[:error] = error
           end
-
-          payload[:process_ids] = executions.map(&:process_id).uniq
-          payload[:job_ids] = executions.map(&:job_id).uniq
-          payload[:size] = executions.size
-          payload[:error] = error
         end
       end
-    end
 
-    def discard_all_in_batches(*)
-      raise UndiscardableError, "Can't discard jobs in progress"
-    end
+      def discard_all_in_batches(*)
+        raise SolidQueue::Execution::UndiscardableError, "Can't discard jobs in progress"
+      end
 
-    def discard_all_from_jobs(*)
-      raise UndiscardableError, "Can't discard jobs in progress"
+      def discard_all_from_jobs(*)
+        raise SolidQueue::Execution::UndiscardableError, "Can't discard jobs in progress"
+      end
     end
-  end
 
-  def perform
-    result = execute
+    def perform
+      result = execute
 
-    if result.success?
-      finished
-    else
-      failed_with(result.error)
-      raise result.error
+      if result.success?
+        finalizing { finished }
+      else
+        finalizing { failed_with(result.error) }
+        raise result.error
+      end
     end
-  end
 
-  def release
-    SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do
-      unless_already_finalized do
-        job.dispatch_bypassing_concurrency_limits
-        destroy!
+    def release
+      SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do
+        unless_already_finalized do
+          job.dispatch_bypassing_concurrency_limits
+          destroy!
+        end
       end
     end
-  end
 
-  def discard
-    raise UndiscardableError, "Can't discard a job in progress"
-  end
-
-  def failed_with(error)
-    finalize { job.failed_with(error) }
-  end
-
-  private
-    def execute
-      ActiveJob::Base.execute(job.arguments.merge("provider_job_id" => job.id))
-      Result.new(true, nil)
-    rescue Exception => e
-      Result.new(false, e)
+    def discard
+      raise UndiscardableError, "Can't discard a job in progress"
     end
 
-    def finished
-      finalize { job.finished! }
+    def failed_with(error)
+      finalize { job.failed_with(error) }
     end
 
-    def finalize
-      finalized = unless_already_finalized do
+    private
+      # A failure here means the job already ran but we couldn't record the
+      # outcome, and the claim is still held by this living worker, where no
+      # recovery can reach it: it's a process problem, not a job problem
+      def finalizing
         yield
-        destroy!
-        true
+      rescue => error
+        raise FinalizationError.new(self, cause: error) if still_claimed?
+
+        raise
       end
 
-      # Unblock the next job outside the finalize transaction so a failure while
-      # releasing the concurrency lock or dispatching the next job can't roll back
-      # a job that already finished or failed. Only the actor that owned and
-      # finalized the claim gets here, so the lock is released exactly once.
-      job.unblock_next_blocked_job if finalized
-    end
+      def execute
+        raise Job::ClassMissingError.for(job) if job.job_class.nil?
 
-    def unless_already_finalized
-      transaction do
-        return false unless self.class.unscoped.lock.find_by(id: id)
+        ActiveJob::Base.execute(job.arguments.merge("provider_job_id" => job.id))
+        Result.new(true, nil)
+      rescue Exception => e
+        Result.new(false, e)
+      end
 
-        yield
+      def finished
+        finalize { job.finished! }
       end
-    end
+
+      def finalize
+        finalized = unless_already_finalized do
+          yield
+          destroy!
+          true
+        end
+
+        # Unblock the next job outside the finalize transaction so a failure while
+        # releasing the concurrency lock or dispatching the next job can't roll back
+        # a job that already finished or failed. Only the actor that owned and
+        # finalized the claim gets here, so the lock is released exactly once.
+        job.unblock_next_blocked_job if finalized
+      end
+
+      def unless_already_finalized
+        transaction do
+          return false unless self.class.unscoped.lock.find_by(id: id)
+
+          yield
+        end
+      end
+
+      def still_claimed?
+        self.class.exists?(id)
+      rescue
+        # If we can't check because the DB is unavailable, assume the claim is
+        # still held so the worker can be stopped and replaced.
+        true
+      end
+  end
 end

data/app/models/solid_queue/execution.rb

@@ -28,13 +28,15 @@ module SolidQueue
       end
 
       def discard_all_in_batches(batch_size: 500)
-        pending = count
+        # Newly unblocked executions must survive later discard batches.
+        last_id = maximum(:id) || 0
+        pending = where(id: ..last_id).count
         discarded = 0
 
         SolidQueue.instrument(:discard_all, batch_size: batch_size, status: type, batches: 0, size: 0) do |payload|
           loop do
             transaction do
-              job_ids = limit(batch_size).order(:job_id).lock.pluck(:job_id)
+              job_ids = where(id: ..last_id).limit(batch_size).order(:job_id).lock.pluck(:job_id)
               discarded = discard_jobs job_ids
 
               where(job_id: job_ids).delete_all

data/app/models/solid_queue/failed_execution.rb

@@ -13,7 +13,10 @@ module SolidQueue
     def self.retry_all(jobs)
       SolidQueue.instrument(:retry_all, jobs_size: jobs.size) do |payload|
         transaction do
-          payload[:size] = dispatch_jobs lock_all_from_jobs(jobs)
+          job_ids = lock_all_from_jobs(jobs)
+          Job.where(id: job_ids).find_each(&:reset_execution_counters)
+
+          payload[:size] = dispatch_jobs(job_ids)
         end
       end
     end
@@ -33,11 +36,11 @@ module SolidQueue
     end
 
     private
-      JSON_OVERHEAD = 256
+      TRUNCATED_MESSAGE_OMISSION = "... (truncated)"
 
       def expand_error_details_from_exception
         if exception
-          self.error = { exception_class: exception_class_name, message: exception_message, backtrace: exception_backtrace }
+          self.error = fit_into_error_column(exception_class: exception_class_name, message: exception_message, backtrace: exception_backtrace)
         end
       end
 
@@ -50,32 +53,58 @@ module SolidQueue
       end
 
       def exception_backtrace
-        if (limit = determine_backtrace_size_limit) && exception.backtrace.to_json.bytesize > limit
-          truncate_backtrace(exception.backtrace, limit)
-        else
-          exception.backtrace
+        exception.backtrace
+      end
+
+      # Columns with a size limit, like MySQL's TEXT, reject (or truncate into invalid JSON)
+      # errors that don't fit. Drop backtrace lines first and, if that's not enough, cut the
+      # message, measuring the error as it'll be stored, after JSON escaping.
+      def fit_into_error_column(error)
+        if (limit = error_column_limit) && serialized_error_size(error) > limit
+          error[:backtrace] = truncate_backtrace(error, limit) if error[:backtrace]
+          error[:message] = truncate_message(error, limit) if serialized_error_size(error) > limit
         end
+
+        error
       end
 
-      def determine_backtrace_size_limit
+      def error_column_limit
         column = self.class.connection_pool.with_connection do |connection|
           connection.schema_cache.columns_hash(self.class.table_name)["error"]
         end
 
-        if column && column.limit.present?
-          column.limit - exception_class_name.bytesize - exception_message.bytesize - JSON_OVERHEAD
+        column&.limit
+      end
+
+      def serialized_error_size(error)
+        self.class.type_for_attribute(:error).serialize(error).bytesize
+      end
+
+      def truncate_backtrace(error, limit)
+        # Each kept line adds its own encoded size plus a separating comma
+        available = limit - serialized_error_size(error.merge(backtrace: [])) + 1
+
+        error[:backtrace].take_while do |line|
+          (available -= serialized_error_size(line) + 1) >= 0
         end
       end
 
-      def truncate_backtrace(lines, limit)
-        [].tap do |truncated_backtrace|
-          lines.each do |line|
-            if (truncated_backtrace << line).to_json.bytesize > limit
-              truncated_backtrace.pop
-              break
-            end
+      def truncate_message(error, limit)
+        message = error[:message].to_s
+
+        # Every byte cut from the message shrinks its encoded size by at least one byte,
+        # so cutting the overflow converges in very few iterations, even with escaping
+        while message.present? && (overflow = serialized_error_size(error.merge(message: message)) - limit) > 0
+          truncate_to = message.bytesize - overflow
+
+          message = if truncate_to > TRUNCATED_MESSAGE_OMISSION.bytesize
+            message.truncate_bytes(truncate_to, omission: TRUNCATED_MESSAGE_OMISSION)
+          else
+            ""
           end
         end
+
+        message
       end
   end
 end

data/app/models/solid_queue/job/concurrency_controls.rb

@@ -14,8 +14,8 @@ module SolidQueue
       end
 
       class_methods do
-        def release_all_concurrency_locks(jobs)
-          Semaphore.signal_all(jobs.select(&:concurrency_limited?))
+        def unblock_next_blocked_jobs(jobs)
+          jobs.find_each(&:unblock_next_blocked_job)
         end
       end
 
@@ -33,6 +33,10 @@ module SolidQueue
         blocked_execution.present?
       end
 
+      def job_class
+        @job_class ||= class_name.safe_constantize
+      end
+
       private
         def concurrency_on_conflict
           job_class.concurrency_on_conflict.to_s.inquiry
@@ -59,17 +63,13 @@ module SolidQueue
         end
 
         def block
-          BlockedExecution.create_or_find_by!(job_id: id)
+          BlockedExecution.create_or_find_by!(job: self)
         end
 
         def release_next_blocked_job
           BlockedExecution.release_one(concurrency_key)
         end
 
-        def job_class
-          @job_class ||= class_name.safe_constantize
-        end
-
         def execution
           super || blocked_execution
         end

data/app/models/solid_queue/job/executable.rb

@@ -39,8 +39,12 @@ module SolidQueue
             ReadyExecution.create_all_from_jobs jobs
           end
 
+          # Each job acquires its concurrency lock here, and holds it until the
+          # transaction dispatching the whole set ends. Going through them by
+          # concurrency key makes everyone dispatching at the same time acquire
+          # the locks they share in the same order, so they can't deadlock.
           def dispatch_all_one_by_one(jobs)
-            jobs.each(&:dispatch)
+            jobs.sort_by { |job| [ job.concurrency_key, job.id ] }.each(&:dispatch)
           end
 
           def successfully_dispatched(jobs)
@@ -106,7 +110,7 @@ module SolidQueue
 
       private
         def ready
-          ReadyExecution.create_or_find_by!(job_id: id)
+          ReadyExecution.create_or_find_by!(job: self)
         end
 
         def execution

data/app/models/solid_queue/job/schedulable.rb

@@ -38,7 +38,7 @@ module SolidQueue
 
       private
         def schedule
-          ScheduledExecution.create_or_find_by!(job_id: id)
+          ScheduledExecution.create_or_find_by!(job: self)
         end
 
         def execution

data/app/models/solid_queue/job.rb

@@ -4,6 +4,15 @@ module SolidQueue
   class Job < Record
     class EnqueueError < StandardError; end
 
+    # Raised when a job's class can't be resolved anymore, typically because it
+    # was renamed or removed in a deploy while jobs referencing it were in
+    # flight. It subclasses NameError, which is what resolving the class raises.
+    class ClassMissingError < NameError
+      def self.for(job)
+        new("Job class #{job.class_name.inspect} could not be resolved")
+      end
+    end
+
     include Executable, Clearable, Recurrable, Batchable
 
     serialize :arguments, coder: JSON

data/app/models/solid_queue/process.rb

@@ -21,10 +21,14 @@ class SolidQueue::Process < SolidQueue::Record
 
   def heartbeat
     # Clear any previous changes before locking, for example, in case a previous heartbeat
-    # failed because of a DB issue (with SQLite depending on configuration, a BusyException
-    # is not rare) and we still have the unpersisted value
+    # failed because of a DB issue and we still have the unpersisted value
     restore_attributes
     with_lock { touch(:last_heartbeat_at) }
+  rescue
+    # touch writes the attribute before persisting; don't let a failed
+    # update leave this object claiming a heartbeat that was never persisted
+    restore_attributes
+    raise
   end
 
   def deregister(pruned: false)

data/app/models/solid_queue/queue_selector.rb

@@ -34,7 +34,7 @@ module SolidQueue
       def eligible_queues
         if include_all_queues? then all_queues
         else
-          in_raw_order(exact_names + prefixed_names)
+          in_raw_order(exact_names | prefixed_names)
         end
       end
 
@@ -53,12 +53,12 @@ module SolidQueue
       def prefixed_names
         if prefixes.empty? then []
         else
-          relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct_values_of(:queue_name)
+          relation.where(([ "queue_name LIKE ? ESCAPE '!'" ] * prefixes.count).join(" OR "), *prefixes).distinct_values_of(:queue_name)
         end
       end
 
       def prefixes
-        @prefixes ||= raw_queues.select { |queue| prefixed_name?(queue) }.map { |queue| queue.tr("*", "%") }
+        @prefixes ||= raw_queues.select { |queue| prefixed_name?(queue) }.map { |queue| relation.sanitize_sql_like(queue, "!").tr("*", "%") }
       end
 
       def exact_name?(queue)
@@ -74,9 +74,9 @@ module SolidQueue
       end
 
       def in_raw_order(queues)
-        # Only need to sort if we have prefixes and more than one queue name.
-        # Exact names are selected in the same order as they're found
-        if queues.one? || prefixes.empty?
+        # Exact names are already in configuration order. Prefix matches always
+        # go through the literal filter so a single false LIKE hit is not returned.
+        if prefixes.empty?
           queues
         else
           queues = queues.dup
@@ -89,9 +89,7 @@ module SolidQueue
           queues.delete(raw_queue)
         elsif prefixed_name?(raw_queue)
           prefix = raw_queue.tr("*", "")
-          queues.select { |queue| queue.start_with?(prefix) }.tap do |matches|
-            queues -= matches
-          end
+          queues.extract! { |queue| queue.start_with?(prefix) }
         end
       end
   end

data/app/models/solid_queue/ready_execution.rb

@@ -45,7 +45,7 @@ module SolidQueue
 
 
         def discard_jobs(job_ids)
-          Job.release_all_concurrency_locks Job.where(id: job_ids)
+          Job.unblock_next_blocked_jobs Job.where(id: job_ids)
           super
         end
     end

data/app/models/solid_queue/semaphore.rb

@@ -14,10 +14,6 @@ module SolidQueue
         Proxy.new(job).signal
       end
 
-      def signal_all(jobs)
-        Proxy.signal_all(jobs)
-      end
-
       # Requires a unique index on key
       def create_unique_by(attributes)
         if supports_insert_conflict_target?
@@ -31,16 +27,6 @@ module SolidQueue
     end
 
     class Proxy
-      def self.signal_all(jobs)
-        # Guard against incrementing a semaphore's value beyond its limit. Jobs can
-        # have different limits, so group them and cap each group with `value < limit`.
-        jobs.group_by { |job| job.concurrency_limit || 1 }.each do |limit, grouped_jobs|
-          Semaphore.where(key: grouped_jobs.map(&:concurrency_key))
-            .where(value: ...limit)
-            .update_all("value = value + 1")
-        end
-      end
-
       def initialize(job)
         @job = job
       end

data/lib/puma/plugin/solid_queue.rb

@@ -33,6 +33,11 @@ Puma::Plugin.create do
           @solid_queue_pid = fork do
             Thread.new { monitor_puma }
             SolidQueue::Supervisor.start(mode: :fork)
+
+            # Same as Processes::Supervised#create_fork: skip at-exit
+            # finalization, which can deadlock on database handles when a
+            # thread was killed while inside a query
+            exit!(0)
           end
         end
 
@@ -43,6 +48,11 @@ Puma::Plugin.create do
           @solid_queue_pid = fork do
             Thread.new { monitor_puma }
             start_solid_queue(mode: :fork)
+
+            # Same as Processes::Supervised#create_fork: skip at-exit
+            # finalization, which can deadlock on database handles when a
+            # thread was killed while inside a query
+            exit!(0)
           end
         end
 
@@ -129,5 +139,9 @@ Puma::Plugin.create do
 
     def log(...)
       log_writer.log(...)
+    rescue Errno::EIO, Errno::EPIPE, Errno::EBADF
+      # The controlling terminal can disappear, or the output pipe close, before
+      # the monitor thread shuts down the child process. Keep shutdown moving
+      # even when logging cannot.
     end
 end

data/lib/solid_queue/async_supervisor.rb

@@ -27,8 +27,7 @@ module SolidQueue
           if (instance = process_instances.delete(thread_id))
             payload[:thread] = instance
 
-            error = Processes::ThreadTerminatedError.new(instance.name)
-            release_claimed_jobs_by(instance, with_error: error)
+            attempt_to_release_claimed_jobs_by(instance, with_error: Processes::ThreadTerminatedError.new(instance.name))
 
             start_process(configured_processes.delete(thread_id))
           end
@@ -39,12 +38,26 @@ module SolidQueue
         process_instances.values.each(&:stop)
 
         Timer.wait_until(SolidQueue.shutdown_timeout, -> { all_processes_terminated? })
+
+        # The exit! that follows skips the shutdown callbacks, so release the jobs
+        # of the threads still running now instead of leaving them to be pruned
+        attempt_to_deregister unless all_processes_terminated?
       end
 
       def perform_immediate_termination
         exit!
       end
 
+      def attempt_to_deregister
+        wrap_in_app_executor do
+          # Not our own copy: it shares the threads' registrations in memory, and
+          # those of threads that already deregistered are destroyed and frozen
+          SolidQueue::Process.find_by(id: process_id)&.deregister
+        end
+      rescue StandardError => error
+        handle_thread_error(error)
+      end
+
       def all_processes_terminated?
         process_instances.values.none?(&:alive?)
       end

data/lib/solid_queue/fiber_pool.rb

@@ -2,7 +2,7 @@
 
 module SolidQueue
   class FiberPool < Pool
-    def initialize(size, on_idle: nil)
+    def initialize(size, on_idle: nil, on_unrecoverable_error: nil)
       super
 
       @state_mutex = Mutex.new
@@ -108,6 +108,7 @@ module SolidQueue
         handle_thread_error(error)
         register_fatal_error(error)
       rescue Exception => error
+        handle_unrecoverable_error(error)
         handle_thread_error(error)
       ensure
         restore_capacity

data/lib/solid_queue/fork_supervisor.rb

@@ -57,7 +57,7 @@ module SolidQueue
           terminated_fork.mark_as_reaped
 
           if !status.exited? || status.exitstatus.to_i > 0
-            attempt_to_release_claimed_jobs_by(terminated_fork, status)
+            attempt_to_release_claimed_jobs_by(terminated_fork, with_error: Processes::ProcessExitError.new(status))
           end
         end
 
@@ -73,23 +73,13 @@ module SolidQueue
           terminated_fork.mark_as_reaped
           payload[:fork] = terminated_fork
 
-          attempt_to_release_claimed_jobs_by(terminated_fork, status)
+          attempt_to_release_claimed_jobs_by(terminated_fork, with_error: Processes::ProcessExitError.new(status))
 
           start_process(configured_processes.delete(pid))
         end
       end
     end
 
-    # The database may be unreachable — likely the same reason the fork
-    # terminated. Neither starting a replacement nor shutting down can depend
-    # on it: the jobs claimed by the terminated fork will be failed when its
-    # stale registration is pruned once the database is back.
-    def attempt_to_release_claimed_jobs_by(terminated_fork, status)
-      release_claimed_jobs_by(terminated_fork, with_error: Processes::ProcessExitError.new(status))
-    rescue StandardError => error
-      handle_thread_error(error)
-    end
-
     def all_processes_terminated?
       process_instances.empty?
     end

data/lib/solid_queue/log_subscriber.rb

@@ -59,7 +59,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
   end
 
   def release_blocked(event)
-    debug formatted_event(event, action: "Release blocked job", **event.payload.slice(:job_id, :concurrency_key, :released))
+    debug formatted_event(event, action: "Release blocked job", **event.payload.slice(:job_id, :concurrency_key, :released, :failed))
   end
 
   def enqueue_recurring_task(event)

data/lib/solid_queue/pool.rb

@@ -4,15 +4,20 @@ module SolidQueue
   class Pool
     include AppExecutor
 
-    def self.build(type:, size:, on_idle: nil)
-      SolidQueue.const_get("#{type.to_s.camelize}Pool").new(size, on_idle: on_idle)
+    def self.build(type:, size:, on_idle: nil, on_unrecoverable_error: nil)
+      SolidQueue.const_get("#{type.to_s.camelize}Pool").new(
+        size,
+        on_idle: on_idle,
+        on_unrecoverable_error: on_unrecoverable_error
+      )
     end
 
     attr_reader :size
 
-    def initialize(size, on_idle: nil)
+    def initialize(size, on_idle: nil, on_unrecoverable_error: nil)
       @size = size
       @on_idle = on_idle
+      @on_unrecoverable_error = on_unrecoverable_error
       @available_capacity = size
       @mutex = Mutex.new
     end
@@ -41,7 +46,7 @@ module SolidQueue
     end
 
     private
-      attr_reader :mutex, :on_idle
+      attr_reader :mutex, :on_idle, :on_unrecoverable_error
 
       def schedule(execution)
         raise NotImplementedError
@@ -50,11 +55,20 @@ module SolidQueue
       def perform_execution(execution)
         wrap_in_app_executor { execution.perform }
       rescue Exception => error
+        handle_unrecoverable_error(error)
         handle_thread_error(error)
       ensure
         restore_capacity
       end
 
+      def handle_unrecoverable_error(error)
+        return unless error.is_a?(Processes::UnrecoverableError)
+
+        # Only signal shutdown — do not join the worker from this pool thread,
+        # or wait_for_termination during worker shutdown would deadlock.
+        on_unrecoverable_error&.call
+      end
+
       def reserve_capacity!
         mutex.synchronize do
           raise RuntimeError, "Execution pool is at capacity" if @available_capacity <= 0

data/lib/solid_queue/processes/registrable.rb

@@ -56,6 +56,26 @@ module SolidQueue::Processes
       def heartbeat
         process&.heartbeat
       rescue ActiveRecord::RecordNotFound
+        # Our registration is gone: a supervisor pruned it
+        stop_to_be_replaced
+      rescue => error
+        # Errors like a dropped database connection prevent the
+        # heartbeat from going through, and even from finding out whether the
+        # registration is still there
+        stop_to_be_replaced if presumed_dead?
+        raise error
+      end
+
+      # Whether this process's registration is prunable: if the last heartbeat that
+      # we were able to persist is older than the alive threshold, supervisors
+      # consider this one dead and would have pruned its registration by now
+      def presumed_dead?
+        process && process.last_heartbeat_at <= SolidQueue.process_alive_threshold.ago
+      end
+
+      # Deregister locally and wake the run loop, which stops when
+      # unregistered, so the supervisor replaces this process
+      def stop_to_be_replaced
         self.process = nil
         wake_up
       end

data/lib/solid_queue/processes/supervised.rb

@@ -29,6 +29,14 @@ module SolidQueue::Processes
         fork do
           register_signal_handlers
           block.call
+
+          # Exit skipping at-exit hooks and finalizers, like Puma's cluster
+          # workers do: Ruby would finalize every object still alive in the
+          # fork, including SQLite database handles whose mutex can be left
+          # locked when a thread is killed while waiting in SQLite's busy
+          # handler, deadlocking the exit. Everything the process needs to do
+          # on shutdown has already run by now.
+          exit!(0)
         end
       end
 

data/lib/solid_queue/processes/unrecoverable_error.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module Processes
+    # Errors that mean the process raising them can no longer account for the
+    # work it holds, so it should stop and let its supervisor replace it
+    class UnrecoverableError < RuntimeError; end
+  end
+end

data/lib/solid_queue/scheduler/recurring_schedule.rb

@@ -10,6 +10,7 @@ module SolidQueue
       @static_tasks = Array(static_tasks).map { |task| RecurringTask.wrap(task) }.select(&:valid?)
       @dynamic_tasks_enabled = dynamic_tasks_enabled
 
+      @schedule_lock = Mutex.new
       @scheduled_tasks = Concurrent::Hash.new
     end
 
@@ -28,18 +29,16 @@ module SolidQueue
         reload_dynamic_tasks
       end
 
-      configured_tasks.each do |task|
-        schedule_task(task)
+      schedule_lock.synchronize do
+        configured_tasks.each { |task| schedule_task(task) }
       end
     end
 
-    def schedule_task(task, run_at: task.next_time)
-      scheduled_tasks[task.key] = schedule(task, run_at: run_at)
-    end
-
     def unschedule_tasks
-      scheduled_tasks.values.each(&:cancel)
-      scheduled_tasks.clear
+      schedule_lock.synchronize do
+        scheduled_tasks.values.each(&:cancel)
+        scheduled_tasks.clear
+      end
     end
 
     def task_keys
@@ -49,13 +48,16 @@ module SolidQueue
     def reschedule_dynamic_tasks
       wrap_in_app_executor do
         reload_dynamic_tasks
-        schedule_created_dynamic_tasks
-        unschedule_deleted_dynamic_tasks
+
+        schedule_lock.synchronize do
+          schedule_created_dynamic_tasks
+          unschedule_deleted_dynamic_tasks
+        end
       end
     end
 
     private
-      attr_reader :static_tasks
+      attr_reader :static_tasks, :schedule_lock
 
       def static_task_keys
         static_tasks.map(&:key)
@@ -69,19 +71,22 @@ module SolidQueue
         @dynamic_tasks_enabled
       end
 
+      # Scheduled tasks are tracked by record id, so a dynamic task that's deleted and
+      # created again with the same key is unscheduled and scheduled as a new one.
       def schedule_created_dynamic_tasks
-        RecurringTask.dynamic.where.not(key: scheduled_tasks.keys).each do |task|
-          schedule_task(task)
-        end
+        dynamic_tasks.reject { |task| scheduled_tasks.key?(task.id) }.each { |task| schedule_task(task) }
       end
 
       def unschedule_deleted_dynamic_tasks
-        (scheduled_tasks.keys - RecurringTask.pluck(:key)).each do |key|
-          scheduled_tasks[key].cancel
-          scheduled_tasks.delete(key)
+        (scheduled_tasks.keys - static_tasks.map(&:id) - dynamic_tasks.map(&:id)).each do |id|
+          scheduled_tasks.delete(id).cancel
         end
       end
 
+      def schedule_task(task, run_at: task.next_time)
+        scheduled_tasks[task.id] = schedule(task, run_at: run_at)
+      end
+
       def persist_static_tasks
         RecurringTask.static.where.not(key: static_task_keys).delete_all
         RecurringTask.create_or_update_all static_tasks
@@ -102,8 +107,8 @@ module SolidQueue
       def schedule(task, run_at: task.next_time)
         delay = [ (run_at - Time.current).to_f, 0.1 ].max
 
-        scheduled_task = Concurrent::ScheduledTask.new(delay, args: [ self, task, run_at ]) do |thread_schedule, thread_task, thread_task_run_at|
-          thread_schedule.schedule_task(thread_task, run_at: thread_task.next_time_after(thread_task_run_at))
+        scheduled_task = Concurrent::ScheduledTask.new(delay, args: [ task, run_at ]) do |thread_task, thread_task_run_at|
+          schedule_next_run(thread_task, run_at: thread_task.next_time_after(thread_task_run_at))
 
           wrap_in_app_executor do
             thread_task.enqueue(at: thread_task_run_at)
@@ -118,5 +123,12 @@ module SolidQueue
 
         scheduled_task.tap(&:execute)
       end
+
+      # Only schedule the next run if the task wasn't unscheduled in the meantime
+      def schedule_next_run(task, run_at:)
+        schedule_lock.synchronize do
+          schedule_task(task, run_at: run_at) if scheduled_tasks.key?(task.id)
+        end
+      end
   end
 end

data/lib/solid_queue/supervisor/maintenance.rb

@@ -43,5 +43,15 @@ module SolidQueue
           end
         end
       end
+
+      # The database may be unreachable — possibly the same reason the
+      # supervised process died. Neither starting a replacement nor shutting
+      # down can depend on it: the jobs claimed by the dead process will be
+      # failed when its stale registration is pruned once the database is back.
+      def attempt_to_release_claimed_jobs_by(terminated_process, with_error:)
+        release_claimed_jobs_by(terminated_process, with_error: with_error)
+      rescue StandardError => error
+        handle_thread_error(error)
+      end
   end
 end

data/lib/solid_queue/thread_pool.rb

@@ -17,6 +17,7 @@ module SolidQueue
         end.on_rejection! do |error|
           # Backstop for errors raised outside perform_execution's own rescue,
           # such as when restoring capacity or waking up the worker
+          handle_unrecoverable_error(error)
           handle_thread_error(error)
         end
       end

data/lib/solid_queue/version.rb

@@ -1,5 +1,5 @@
 module SolidQueue
-  VERSION = "1.7.0"
+  VERSION = "1.7.1"
 
   def self.next_major_version
     Gem::Version.new(VERSION).segments.first + 1

data/lib/solid_queue/worker.rb

@@ -22,7 +22,8 @@ module SolidQueue
       @pool = Pool.build \
         type: execution_pool_type,
         size: execution_pool_size,
-        on_idle: -> { wake_up }
+        on_idle: -> { wake_up },
+        on_unrecoverable_error: -> { request_termination }
 
       super(**options)
     end
@@ -48,6 +49,14 @@ module SolidQueue
         end
       end
 
+      def request_termination
+        # Signal the poller to shut down without joining from the pool thread.
+        # Runnable#stop joins when unsupervised, which would deadlock once
+        # shutdown waits for this pool thread to finish.
+        @stopped = true
+        wake_up
+      end
+
       def shutdown
         pool.shutdown
         pool.wait_for_termination(SolidQueue.shutdown_timeout)

data/lib/solid_queue.rb

@@ -16,6 +16,8 @@ loader = Zeitwerk::Loader.for_gem(warn_on_extra_files: false)
 loader.ignore("#{__dir__}/solid_queue/tasks.rb")
 loader.ignore("#{__dir__}/generators")
 loader.ignore("#{__dir__}/puma")
+# Only bin/jobs uses the CLI, and requires it. Eager loading it would load Thor in every process.
+loader.do_not_eager_load("#{__dir__}/solid_queue/cli.rb")
 loader.setup
 
 module SolidQueue

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 1.7.0
+  version: 1.7.1
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-08-21 00:00:00.000000000 Z
+date: 2026-11-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -361,6 +361,7 @@ files:
 - lib/solid_queue/processes/runnable.rb
 - lib/solid_queue/processes/supervised.rb
 - lib/solid_queue/processes/thread_terminated_error.rb
+- lib/solid_queue/processes/unrecoverable_error.rb
 - lib/solid_queue/scheduler.rb
 - lib/solid_queue/scheduler/recurring_schedule.rb
 - lib/solid_queue/supervisor.rb