solid_queue 1.1.0 → 1.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e843e842397e1d8141e0457b5b278026b71effe6ebf83d8300d2c4098db56adf
-  data.tar.gz: a5db37bdea8dacec796f2dcb912ab8f2a69c929487dff081d8a53fe10c20086c
+  metadata.gz: 778d9495c79b9b8af00416fd1980250fe6c3213043cbfd31eba5eb5e2795ad13
+  data.tar.gz: 14f929171d65334300648a5f80e59b9f8245034119cfd436d37dae545210c97b
 SHA512:
-  metadata.gz: a6831f7114c24d68ae8ed7b5aebfd4d3df0cc2c7b4e0046e6275a5bf5e46f8da8f5b79e9a303ae908e3581e8b96a132776f4c1a3f55bbf550dd008541e679665
-  data.tar.gz: e51b90234b3a60355a96c9163bfc7d95a1f5eb95fcf7a2dc1faf553f04a7c242da2d48493ba5b6574c2dbf4b0d259cb69af1dda576475f1510a31f8325be5f8b
+  metadata.gz: cbbd8113e179c49ffcfe707aa38c96f31ac812822d380e7a6739daced30ee029046ec19cf2ea70c81e19c0f1a819412454f1f4eb1a4cfad7cf5fa8b0f88e3021
+  data.tar.gz: c324b1127bbad96191d3c2e268058dd1cec7c29cffe45a511b0899f3e028e86b7ad0cf0638c93100393316bde1e429ab28c828a381c0692b8558c100f296b6f6

data/README.md

@@ -6,6 +6,32 @@ Besides regular job enqueuing and processing, Solid Queue supports delayed jobs,
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails's multi-threading.
 
+## Table of contents
+
+- [Installation](#installation)
+  - [Usage in development and other non-production environments](#usage-in-development-and-other-non-production-environments)
+  - [Single database configuration](#single-database-configuration)
+  - [Incremental adoption](#incremental-adoption)
+  - [High performance requirements](#high-performance-requirements)
+- [Configuration](#configuration)
+  - [Workers, dispatchers and scheduler](#workers-dispatchers-and-scheduler)
+  - [Queue order and priorities](#queue-order-and-priorities)
+  - [Queues specification and performance](#queues-specification-and-performance)
+  - [Threads, processes and signals](#threads-processes-and-signals)
+  - [Database configuration](#database-configuration)
+  - [Other configuration settings](#other-configuration-settings)
+- [Lifecycle hooks](#lifecycle-hooks)
+- [Errors when enqueuing](#errors-when-enqueuing)
+- [Concurrency controls](#concurrency-controls)
+- [Failed jobs and retries](#failed-jobs-and-retries)
+  - [Error reporting on jobs](#error-reporting-on-jobs)
+- [Puma plugin](#puma-plugin)
+- [Jobs and transactional integrity](#jobs-and-transactional-integrity)
+- [Recurring tasks](#recurring-tasks)
+- [Inspiration](#inspiration)
+- [License](#license)
+
+
 ## Installation
 
 Solid Queue is configured by default in new Rails 8 applications. But if you're running an earlier version, you can add it manually following these steps:
@@ -13,6 +39,8 @@ Solid Queue is configured by default in new Rails 8 applications. But if you're
 1. `bundle add solid_queue`
 2. `bin/rails solid_queue:install`
 
+(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.1.6.)
+
 This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue.
 
 Once you've done that, you will then have to add the configuration for the queue database in `config/database.yml`. If you're using SQLite, it'll look like this:
@@ -43,8 +71,6 @@ production:
     migrations_paths: db/queue_migrate
 ```
 
-Note: Calling `bin/rails solid_queue:install` will automatically add `config.solid_queue.connects_to = { database: { writing: :queue } }` to `config/environments/production.rb`, so no additional configuration is needed there (although you must make sure that you use the `queue` name in `database.yml` for this to match!). But if you want to use Solid Queue in a different environment (like staging or even development), you'll have to manually add that `config.solid_queue.connects_to` line to the respective environment file. And, as always, make sure that the name you're using for the database in `config/database.yml` matches the name you use in `config.solid_queue.connects_to`.
-
 Then run `db:prepare` in production to ensure the database is created and the schema is loaded.
 
 Now you're ready to start processing jobs by running `bin/jobs` on the server that's doing the work. This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
@@ -53,6 +79,72 @@ For small projects, you can run Solid Queue on the same machine as your webserve
 
 **Note**: future changes to the schema will come in the form of regular migrations.
 
+### Usage in development and other non-production environments
+
+Calling `bin/rails solid_queue:install` will automatically add `config.solid_queue.connects_to = { database: { writing: :queue } }` to `config/environments/production.rb`. In order to use Solid Queue in other environments (such as development or staging), you'll need to add a similar configuration(s).
+
+For example, if you're using SQLite in development, update `database.yml` as follows:
+
+```diff
+development:
++ primary:
+    <<: *default
+    database: storage/development.sqlite3
++  queue:
++    <<: *default
++    database: storage/development_queue.sqlite3
++    migrations_paths: db/queue_migrate
+```
+
+Next, add the following to `development.rb`
+
+```ruby
+  # Use Solid Queue in Development.
+  config.active_job.queue_adapter = :solid_queue
+  config.solid_queue.connects_to = { database: { writing: :queue } }
+```
+
+Once you've added this, run `db:prepare` to create the Solid Queue database and load the schema.
+
+Finally, in order for jobs to be processed, you'll need to have Solid Queue running. In Development, this can be done via [the Puma plugin](#puma-plugin) as well. In `puma.rb` update the following line:
+
+```ruby
+# You can either set the env var, or check for development
+plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"] || Rails.env.development?
+```
+
+You can also just use `bin/jobs`, but in this case you might want to [set a different logger for Solid Queue](#other-configuration-settings) because the default logger will log to `log/development.log` and you won't see anything when you run `bin/jobs`. For example:
+```ruby
+config.solid_queue.logger = ActiveSupport::Logger.new(STDOUT)
+```
+
+**Note about Action Cable**: If you use Action Cable (or anything dependent on Action Cable, such as Turbo Streams), you will also need to update it to use a database.
+
+In `config/cable.yml`
+
+```diff
+development:
+-  adapter: async
++ adapter: solid_cable
++  connects_to:
++    database:
++      writing: cable
++  polling_interval: 0.1.seconds
++  message_retention: 1.day
+```
+
+In `config/database.yml`
+
+```diff
+development:
+  primary:
+    <<: *default
+    database: storage/development.sqlite3
++  cable:
++    <<: *default
++    database: storage/development_cable.sqlite3
++    migrations_paths: db/cable_migrate
+```
 
 ### Single database configuration
 
@@ -64,7 +156,7 @@ Running Solid Queue in a separate database is recommended, but it's also possibl
 
 You won't have multiple databases, so `database.yml` doesn't need to have primary and queue database.
 
-## Incremental adoption
+### Incremental adoption
 
 If you're planning to adopt Solid Queue incrementally by switching one job at the time, you can do so by leaving the `config.active_job.queue_adapter` set to your old backend, and then set the `queue_adapter` directly in the jobs you're moving:
 
@@ -77,7 +169,7 @@ class MyJob < ApplicationJob
 end
 ```
 
-## High performance requirements
+### High performance requirements
 
 Solid Queue was designed for the highest throughput when used with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue. You can also use it with SQLite on smaller applications.
 
@@ -86,6 +178,7 @@ Solid Queue was designed for the highest throughput when used with MySQL 8+ or P
 ### Workers, dispatchers and scheduler
 
 We have several types of actors in Solid Queue:
+
 - _Workers_ are in charge of picking jobs ready to run from queues and processing them. They work off the `solid_queue_ready_executions` table.
 - _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
 - The _scheduler_ manages [recurring tasks](#recurring-tasks), enqueuing jobs for them when they're due.
@@ -99,7 +192,6 @@ By default, Solid Queue will try to find your configuration under `config/queue.
 bin/jobs -c config/calendar.yml
 ```
 
-
 This is what this configuration looks like:
 
 ```yml
@@ -153,6 +245,7 @@ Here's an overview of the different options:
   Check the sections below on [how queue order behaves combined with priorities](#queue-order-and-priorities), and [how the way you specify the queues per worker might affect performance](#queues-specification-and-performance).
 
 - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
+It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker thread uses one connection, and two additional connections are reserved for polling and heartbeat.
 - `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 
@@ -220,7 +313,7 @@ and then remove the paused ones. Pausing in general should be something rare, us
 Do this:
 
 ```yml
-queues: background, backend
+queues: [ background, backend ]
 ```
 
 instead of this:
@@ -250,33 +343,6 @@ You can configure the database used by Solid Queue via the `config.solid_queue.c
 
 All the options available to Active Record for multiple databases can be used here.
 
-## Lifecycle hooks
-
-In Solid queue, you can hook into two different points in the supervisor's life:
-- `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
-- `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
-
-And into two different points in a worker's life:
-- `worker_start`: after the worker has finished booting and right before it starts the polling loop.
-- `worker_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
-
-You can use the following methods with a block to do this:
-```ruby
-SolidQueue.on_start
-SolidQueue.on_stop
-
-SolidQueue.on_worker_start
-SolidQueue.on_worker_stop
-```
-
-For example:
-```ruby
-SolidQueue.on_start { start_metrics_server }
-SolidQueue.on_stop { stop_metrics_server }
-```
-
-These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
-
 ### Other configuration settings
 
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
@@ -299,9 +365,58 @@ There are several settings that control how Solid Queue works that you can set a
 - `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to `true`.
 - `supervisor_pidfile`: path to a pidfile that the supervisor will create when booting to prevent running more than one supervisor in the same host, or in case you want to use it for a health check. It's `nil` by default.
 - `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to `true`.
-- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, but this will happen automatically in the near future.
+- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, which can be configured as [a recurring task](#recurring-tasks).
 - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
 
+
+## Lifecycle hooks
+
+In Solid queue, you can hook into two different points in the supervisor's life:
+- `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
+- `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
+
+And into two different points in the worker's, dispatcher's and scheduler's life:
+- `(worker|dispatcher|scheduler)_start`: after the worker/dispatcher/scheduler has finished booting and right before it starts the polling loop or loading the recurring schedule.
+- `(worker|dispatcher|scheduler)_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
+
+Each of these hooks has an instance of the supervisor/worker/dispatcher/scheduler yielded to the block so that you may read its configuration for logging or metrics reporting purposes.
+
+You can use the following methods with a block to do this:
+```ruby
+SolidQueue.on_start
+SolidQueue.on_stop
+
+SolidQueue.on_worker_start
+SolidQueue.on_worker_stop
+
+SolidQueue.on_dispatcher_start
+SolidQueue.on_dispatcher_stop
+
+SolidQueue.on_scheduler_start
+SolidQueue.on_scheduler_stop
+```
+
+For example:
+```ruby
+SolidQueue.on_start do |supervisor|
+  MyMetricsReporter.process_name = supervisor.name
+
+  start_metrics_server
+end
+
+SolidQueue.on_stop do |_supervisor|
+  stop_metrics_server
+end
+
+SolidQueue.on_worker_start do |worker|
+  MyMetricsReporter.process_name = worker.name
+  MyMetricsReporter.queues = worker.queues.join(',')
+end
+```
+
+These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
+
+
 ## Errors when enqueuing
 
 Solid Queue will raise a `SolidQueue::Job::EnqueueError` for any Active Record errors that happen when enqueuing a job. The reason for not raising `ActiveJob::EnqueueError` is that this one gets handled by Active Job, causing `perform_later` to return `false` and set `job.enqueue_error`, yielding the job to a block that you need to pass to `perform_later`. This works very well for your own jobs, but makes failure very hard to handle for jobs enqueued by Rails or other gems, such as `Turbo::Streams::BroadcastJob` or `ActiveStorage::AnalyzeJob`, because you don't control the call to `perform_later` in that cases.
@@ -412,6 +527,12 @@ plugin :solid_queue
 ```
 to your `puma.rb` configuration.
 
+If you're using Puma in development but you don't want to use Solid Queue in development, make sure you avoid the plugin being used, for example using an environment variable like this:
+```ruby
+plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]
+```
+that you set in production only. This is what Rails 8's default Puma config looks like. Otherwise, if you're using Puma in development but not Solid Queue, starting Puma would start also Solid Queue supervisor and it'll most likely fail because it won't be properly configured.
+
 
 ## Jobs and transactional integrity
 :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed and vice versa, and ensuring that your job won't be enqueued until the transaction within which you're enqueuing it is committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you. Because this can be quite tricky and many people shouldn't need to worry about it, by default Solid Queue is configured in a different database as the main app.
@@ -477,9 +598,15 @@ MyJob.perform_later(42, status: "custom_status")
 
 - `priority`: a numeric priority value to be used when enqueuing the job.
 
-
 Tasks are enqueued at their corresponding times by the scheduler, and each task schedules the next one. This is pretty much [inspired by what GoodJob does](https://github.com/bensheldon/good_job/blob/994ecff5323bf0337e10464841128fda100750e6/lib/good_job/cron_manager.rb).
 
+For recurring tasks defined as a `command`, you can also change the job class that runs them as follows:
+```ruby
+Rails.application.config.after_initialize do # or to_prepare
+  SolidQueue::RecurringTask.default_job_class = MyRecurringCommandJob
+end
+```
+
 It's possible to run multiple schedulers with the same `recurring_tasks` configuration, for example, if you have multiple servers for redundancy, and you run the `scheduler` in more than one of them. To avoid enqueuing duplicate tasks at the same time, an entry in a new `solid_queue_recurring_executions` table is created in the same transaction as the job is enqueued. This table has a unique index on `task_key` and `run_at`, ensuring only one entry per task per time will be created. This only works if you have `preserve_finished_jobs` set to `true` (the default), and the guarantee applies as long as you keep the jobs around.
 
 **Note**: a single recurring schedule is supported, so you can have multiple schedulers using the same schedule, but not multiple schedulers using different configurations.

data/app/models/solid_queue/claimed_execution.rb

@@ -92,7 +92,7 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
 
   private
     def execute
-      ActiveJob::Base.execute(job.arguments)
+      ActiveJob::Base.execute(job.arguments.merge("provider_job_id" => job.id))
       Result.new(true, nil)
     rescue Exception => e
       Result.new(false, e)

data/app/models/solid_queue/job/clearable.rb

@@ -10,9 +10,10 @@ module SolidQueue
       end
 
       class_methods do
-        def clear_finished_in_batches(batch_size: 500, finished_before: SolidQueue.clear_finished_jobs_after.ago, class_name: nil)
+        def clear_finished_in_batches(batch_size: 500, finished_before: SolidQueue.clear_finished_jobs_after.ago, class_name: nil, sleep_between_batches: 0)
           loop do
             records_deleted = clearable(finished_before: finished_before, class_name: class_name).limit(batch_size).delete_all
+            sleep(sleep_between_batches) if sleep_between_batches > 0
             break if records_deleted == 0
           end
         end

data/app/models/solid_queue/recurring_task.rb

@@ -12,6 +12,8 @@ module SolidQueue
 
     scope :static, -> { where(static: true) }
 
+    has_many :recurring_executions, foreign_key: :task_key, primary_key: :key
+
     mattr_accessor :default_job_class
     self.default_job_class = RecurringJob
 
@@ -53,6 +55,18 @@ module SolidQueue
       parsed_schedule.next_time.utc
     end
 
+    def previous_time
+      parsed_schedule.previous_time.utc
+    end
+
+    def last_enqueued_time
+      if recurring_executions.loaded?
+        recurring_executions.map(&:run_at).max
+      else
+        recurring_executions.maximum(:run_at)
+      end
+    end
+
     def enqueue(at:)
       SolidQueue.instrument(:enqueue_recurring_task, task: key, at: at) do |payload|
         active_job = if using_solid_queue_adapter?

data/app/models/solid_queue/scheduled_execution.rb

@@ -14,7 +14,7 @@ module SolidQueue
       def dispatch_next_batch(batch_size)
         transaction do
           job_ids = next_batch(batch_size).non_blocking_lock.pluck(:job_id)
-          if job_ids.empty? then []
+          if job_ids.empty? then 0
           else
             SolidQueue.instrument(:dispatch_scheduled, batch_size: batch_size) do |payload|
               payload[:size] = dispatch_jobs(job_ids)

data/lib/solid_queue/app_executor.rb

@@ -4,7 +4,7 @@ module SolidQueue
   module AppExecutor
     def wrap_in_app_executor(&block)
       if SolidQueue.app_executor
-        SolidQueue.app_executor.wrap(&block)
+        SolidQueue.app_executor.wrap(source: "application.solid_queue", &block)
       else
         yield
       end

data/lib/solid_queue/configuration.rb

@@ -2,6 +2,12 @@
 
 module SolidQueue
   class Configuration
+    include ActiveModel::Model
+
+    validate :ensure_configured_processes
+    validate :ensure_valid_recurring_tasks
+    validate :ensure_correctly_sized_thread_pool
+
     class Process < Struct.new(:kind, :attributes)
       def instantiate
         "SolidQueue::#{kind.to_s.titleize}".safe_constantize.new(**attributes)
@@ -36,14 +42,46 @@ module SolidQueue
       end
     end
 
-    def max_number_of_threads
-      # At most "threads" in each worker + 1 thread for the worker + 1 thread for the heartbeat task
-      workers_options.map { |options| options[:threads] }.max + 2
+    def error_messages
+      if configured_processes.none?
+        "No workers or processed configured. Exiting..."
+      else
+        error_messages = invalid_tasks.map do |task|
+            all_messages = task.errors.full_messages.map { |msg| "\t#{msg}" }.join("\n")
+            "#{task.key}:\n#{all_messages}"
+          end
+          .join("\n")
+
+        "Invalid processes configured:\n#{error_messages}"
+      end
     end
 
     private
       attr_reader :options
 
+      def ensure_configured_processes
+        unless configured_processes.any?
+          errors.add(:base, "No processes configured")
+        end
+      end
+
+      def ensure_valid_recurring_tasks
+        unless skip_recurring_tasks? || invalid_tasks.none?
+          error_messages = invalid_tasks.map do |task|
+            "- #{task.key}: #{task.errors.full_messages.join(", ")}"
+          end
+
+          errors.add(:base, "Invalid recurring tasks:\n#{error_messages.join("\n")}")
+        end
+      end
+
+      def ensure_correctly_sized_thread_pool
+        if (db_pool_size = SolidQueue::Record.connection_pool&.size) && db_pool_size < estimated_number_of_threads
+          errors.add(:base, "Solid Queue is configured to use #{estimated_number_of_threads} threads but the " +
+            "database connection pool is #{db_pool_size}. Increase it in `config/database.yml`")
+        end
+      end
+
       def default_options
         {
           config_file: Rails.root.join(ENV["SOLID_QUEUE_CONFIG"] || DEFAULT_CONFIG_FILE_PATH),
@@ -54,6 +92,10 @@ module SolidQueue
         }
       end
 
+      def invalid_tasks
+        recurring_tasks.select(&:invalid?)
+      end
+
       def only_work?
         options[:only_work]
       end
@@ -99,8 +141,8 @@ module SolidQueue
 
       def recurring_tasks
         @recurring_tasks ||= recurring_tasks_config.map do |id, options|
-          RecurringTask.from_configuration(id, **options)
-        end.select(&:valid?)
+          RecurringTask.from_configuration(id, **options) if options&.has_key?(:schedule)
+        end.compact
       end
 
       def processes_config
@@ -111,7 +153,9 @@ module SolidQueue
       end
 
       def recurring_tasks_config
-        @recurring_tasks_config ||= config_from options[:recurring_schedule_file]
+        @recurring_tasks_config ||= begin
+          config_from options[:recurring_schedule_file]
+        end
       end
 
 
@@ -147,5 +191,11 @@ module SolidQueue
           {}
         end
       end
+
+      def estimated_number_of_threads
+        # At most "threads" in each worker + 1 thread for the worker + 1 thread for the heartbeat task
+        thread_count = workers_options.map { |options| options.fetch(:threads, WORKER_DEFAULTS[:threads]) }.max
+        (thread_count || 1) + 2
+      end
   end
 end

data/lib/solid_queue/dispatcher.rb

@@ -2,10 +2,14 @@
 
 module SolidQueue
   class Dispatcher < Processes::Poller
-    attr_accessor :batch_size, :concurrency_maintenance
+    include LifecycleHooks
+    attr_reader :batch_size
 
+    after_boot :run_start_hooks
     after_boot :start_concurrency_maintenance
     before_shutdown :stop_concurrency_maintenance
+    before_shutdown :run_stop_hooks
+    after_shutdown :run_exit_hooks
 
     def initialize(**options)
       options = options.dup.with_defaults(SolidQueue::Configuration::DISPATCHER_DEFAULTS)
@@ -22,9 +26,12 @@ module SolidQueue
     end
 
     private
+      attr_reader :concurrency_maintenance
+
       def poll
         batch = dispatch_next_batch
-        batch.size
+
+        batch.zero? ? polling_interval : 0.seconds
       end
 
       def dispatch_next_batch
@@ -37,20 +44,12 @@ module SolidQueue
         concurrency_maintenance&.start
       end
 
-      def schedule_recurring_tasks
-        recurring_schedule.schedule_tasks
-      end
-
       def stop_concurrency_maintenance
         concurrency_maintenance&.stop
       end
 
-      def unschedule_recurring_tasks
-        recurring_schedule.unschedule_tasks
-      end
-
       def all_work_completed?
-        SolidQueue::ScheduledExecution.none? && recurring_schedule.empty?
+        SolidQueue::ScheduledExecution.none?
       end
 
       def set_procline

data/lib/solid_queue/engine.rb

@@ -37,5 +37,13 @@ module SolidQueue
         include ActiveJob::ConcurrencyControls
       end
     end
+
+    initializer "solid_queue.include_interruptible_concern" do
+      if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.2")
+        SolidQueue::Processes::Base.include SolidQueue::Processes::Interruptible
+      else
+        SolidQueue::Processes::Base.include SolidQueue::Processes::OgInterruptible
+      end
+    end
   end
 end

data/lib/solid_queue/lifecycle_hooks.rb

@@ -5,7 +5,7 @@ module SolidQueue
     extend ActiveSupport::Concern
 
     included do
-      mattr_reader :lifecycle_hooks, default: { start: [], stop: [] }
+      mattr_reader :lifecycle_hooks, default: { start: [], stop: [], exit: [] }
     end
 
     class_methods do
@@ -17,7 +17,12 @@ module SolidQueue
         self.lifecycle_hooks[:stop] << block
       end
 
+      def on_exit(&block)
+        self.lifecycle_hooks[:exit] << block
+      end
+
       def clear_hooks
+        self.lifecycle_hooks[:exit] = []
         self.lifecycle_hooks[:start] = []
         self.lifecycle_hooks[:stop] = []
       end
@@ -32,9 +37,13 @@ module SolidQueue
         run_hooks_for :stop
       end
 
+      def run_exit_hooks
+        run_hooks_for :exit
+      end
+
       def run_hooks_for(event)
         self.class.lifecycle_hooks.fetch(event, []).each do |block|
-          block.call
+            block.call(self)
         rescue Exception => exception
           handle_thread_error(exception)
         end

data/lib/solid_queue/log_subscriber.rb

@@ -145,6 +145,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
   end
 
   def replace_fork(event)
+    supervisor_pid = event.payload[:supervisor_pid]
     status = event.payload[:status]
     attributes = event.payload.slice(:pid).merge \
       status: (status.exitstatus || "no exit status set"),
@@ -155,7 +156,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
 
     if replaced_fork = event.payload[:fork]
       info formatted_event(event, action: "Replaced terminated #{replaced_fork.kind}", **attributes.merge(hostname: replaced_fork.hostname, name: replaced_fork.name))
-    else
+    elsif supervisor_pid != 1 # Running Docker, possibly having some processes that have been reparented
       warn formatted_event(event, action: "Tried to replace forked process but it had already died", **attributes)
     end
   end

data/lib/solid_queue/pool.rb

@@ -18,20 +18,16 @@ module SolidQueue
     def post(execution)
       available_threads.decrement
 
-      future = Concurrent::Future.new(args: [ execution ], executor: executor) do |thread_execution|
+      Concurrent::Promises.future_on(executor, execution) do |thread_execution|
         wrap_in_app_executor do
           thread_execution.perform
         ensure
           available_threads.increment
           mutex.synchronize { on_idle.try(:call) if idle? }
         end
+      end.on_rejection! do |e|
+        handle_thread_error(e)
       end
-
-      future.add_observer do |_, _, error|
-        handle_thread_error(error) if error
-      end
-
-      future.execute
     end
 
     def idle_threads

data/lib/solid_queue/processes/base.rb

@@ -4,7 +4,7 @@ module SolidQueue
   module Processes
     class Base
       include Callbacks # Defines callbacks needed by other concerns
-      include AppExecutor, Registrable, Interruptible, Procline
+      include AppExecutor, Registrable, Procline
 
       attr_reader :name
 

data/lib/solid_queue/processes/interruptible.rb

@@ -2,6 +2,8 @@
 
 module SolidQueue::Processes
   module Interruptible
+    include SolidQueue::AppExecutor
+
     def wake_up
       interrupt
     end
@@ -12,14 +14,20 @@ module SolidQueue::Processes
         queue << true
       end
 
+      # Sleeps for 'time'.  Can be interrupted asynchronously and return early via wake_up.
+      # @param time [Numeric, Duration] the time to sleep. 0 returns immediately.
       def interruptible_sleep(time)
-        # Invoking from the main thread can result in a 35% slowdown (at least when running the test suite).
-        # Using some form of Async (Futures) addresses this performance issue.
+        # Invoking this from the main thread may result in significant slowdown.
+        # Utilizing asynchronous execution (Futures) addresses this performance issue.
         Concurrent::Promises.future(time) do |timeout|
-          if timeout > 0 && queue.pop(timeout:)
-            queue.clear
-          end
+          queue.clear unless queue.pop(timeout:).nil?
+        end.on_rejection! do |e|
+          wrapped_exception = RuntimeError.new("Interruptible#interruptible_sleep - #{e.class}: #{e.message}")
+          wrapped_exception.set_backtrace(e.backtrace)
+          handle_thread_error(wrapped_exception)
         end.value
+
+        nil
       end
 
       def queue

data/lib/solid_queue/processes/og_interruptible.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module SolidQueue::Processes
+  # The original implementation of Interruptible that works
+  # with Ruby 3.1 and earlier
+  module OgInterruptible
+    def wake_up
+      interrupt
+    end
+
+    private
+      SELF_PIPE_BLOCK_SIZE = 11
+
+      def interrupt
+        self_pipe[:writer].write_nonblock(".")
+      rescue Errno::EAGAIN, Errno::EINTR
+        # Ignore writes that would block and retry
+        # if another signal arrived while writing
+        retry
+      end
+
+      def interruptible_sleep(time)
+        if time > 0 && self_pipe[:reader].wait_readable(time)
+          loop { self_pipe[:reader].read_nonblock(SELF_PIPE_BLOCK_SIZE) }
+        end
+      rescue Errno::EAGAIN, Errno::EINTR
+      end
+
+      # Self-pipe for signal-handling (http://cr.yp.to/docs/selfpipe.html)
+      def self_pipe
+        @self_pipe ||= create_self_pipe
+      end
+
+      def create_self_pipe
+        reader, writer = IO.pipe
+        { reader: reader, writer: writer }
+      end
+  end
+end

data/lib/solid_queue/processes/poller.rb

@@ -25,11 +25,11 @@ module SolidQueue::Processes
         loop do
           break if shutting_down?
 
-          wrap_in_app_executor do
-            unless poll > 0
-              interruptible_sleep(polling_interval)
-            end
+          delay = wrap_in_app_executor do
+            poll
           end
+
+          interruptible_sleep(delay)
         end
       ensure
         SolidQueue.instrument(:shutdown_process, process: self) do

data/lib/solid_queue/processes/process_pruned_error.rb

@@ -4,7 +4,7 @@ module SolidQueue
   module Processes
     class ProcessPrunedError < RuntimeError
       def initialize(last_heartbeat_at)
-        super("Process was found dead and pruned (last heartbeat at: #{last_heartbeat_at}")
+        super("Process was found dead and pruned (last heartbeat at: #{last_heartbeat_at})")
       end
     end
   end

data/lib/solid_queue/scheduler.rb

@@ -3,11 +3,15 @@
 module SolidQueue
   class Scheduler < Processes::Base
     include Processes::Runnable
+    include LifecycleHooks
 
-    attr_accessor :recurring_schedule
+    attr_reader :recurring_schedule
 
+    after_boot :run_start_hooks
     after_boot :schedule_recurring_tasks
     before_shutdown :unschedule_recurring_tasks
+    before_shutdown :run_stop_hooks
+    after_shutdown :run_exit_hooks
 
     def initialize(recurring_tasks:, **options)
       @recurring_schedule = RecurringSchedule.new(recurring_tasks)

data/lib/solid_queue/supervisor.rb

@@ -5,15 +5,17 @@ module SolidQueue
     include LifecycleHooks
     include Maintenance, Signals, Pidfiled
 
+    after_shutdown :run_exit_hooks
+
     class << self
       def start(**options)
         SolidQueue.supervisor = true
         configuration = Configuration.new(**options)
 
-        if configuration.configured_processes.any?
+        if configuration.valid?
           new(configuration).tap(&:start)
         else
-          abort "No workers or processed configured. Exiting..."
+          abort configuration.errors.full_messages.join("\n") + "\nExiting..."
         end
       end
     end

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "1.1.0"
+  VERSION = "1.1.4"
 end

data/lib/solid_queue/worker.rb

@@ -6,13 +6,16 @@ module SolidQueue
 
     after_boot :run_start_hooks
     before_shutdown :run_stop_hooks
+    after_shutdown :run_exit_hooks
 
-    attr_accessor :queues, :pool
+    attr_reader :queues, :pool
 
     def initialize(**options)
       options = options.dup.with_defaults(SolidQueue::Configuration::WORKER_DEFAULTS)
 
-      @queues = Array(options[:queues])
+      # Ensure that the queues array is deep frozen to prevent accidental modification
+      @queues = Array(options[:queues]).map(&:freeze).freeze
+
       @pool = Pool.new(options[:threads], on_idle: -> { wake_up })
 
       super(**options)
@@ -29,7 +32,7 @@ module SolidQueue
             pool.post(execution)
           end
 
-          executions.size
+          pool.idle? ? polling_interval : 10.minutes
         end
       end
 

data/lib/solid_queue.rb

@@ -41,14 +41,20 @@ module SolidQueue
   mattr_accessor :clear_finished_jobs_after, default: 1.day
   mattr_accessor :default_concurrency_control_period, default: 3.minutes
 
-  delegate :on_start, :on_stop, to: Supervisor
+  delegate :on_start, :on_stop, :on_exit, to: Supervisor
 
-  def on_worker_start(...)
-    Worker.on_start(...)
-  end
+  [ Dispatcher, Scheduler, Worker ].each do |process|
+    define_singleton_method(:"on_#{process.name.demodulize.downcase}_start") do |&block|
+      process.on_start(&block)
+    end
+
+    define_singleton_method(:"on_#{process.name.demodulize.downcase}_stop") do |&block|
+      process.on_stop(&block)
+    end
 
-  def on_worker_stop(...)
-    Worker.on_stop(...)
+    define_singleton_method(:"on_#{process.name.demodulize.downcase}_exit") do |&block|
+      process.on_exit(&block)
+    end
   end
 
   def supervisor?

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.1.4
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-12-05 00:00:00.000000000 Z
+date: 2025-03-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -220,6 +220,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: zeitwerk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.6.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.6.0
 description: Database-backed Active Job backend.
 email:
 - rosa@37signals.com
@@ -281,6 +295,7 @@ files:
 - lib/solid_queue/processes/base.rb
 - lib/solid_queue/processes/callbacks.rb
 - lib/solid_queue/processes/interruptible.rb
+- lib/solid_queue/processes/og_interruptible.rb
 - lib/solid_queue/processes/poller.rb
 - lib/solid_queue/processes/process_exit_error.rb
 - lib/solid_queue/processes/process_missing_error.rb
@@ -307,15 +322,8 @@ metadata:
   homepage_uri: https://github.com/rails/solid_queue
   source_code_uri: https://github.com/rails/solid_queue
 post_install_message: |
-  Upgrading to Solid Queue 0.9.0? There are some breaking changes about how recurring tasks are configured.
-
-  Upgrading to Solid Queue 0.8.0 from < 0.6.0? You need to upgrade to 0.6.0 first.
-
-  Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
-  configuration and new migrations.
-
-  --> Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
-  for upgrade instructions.
+  Upgrading from Solid Queue < 1.0? Check details on breaking changes and upgrade instructions
+  --> https://github.com/rails/solid_queue/blob/main/UPGRADING.md
 rdoc_options: []
 require_paths:
 - lib
@@ -323,7 +331,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '3.1'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="