solid_queue 1.1.2 → 1.2.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 17a05d42c432fb64bc429ef13665309e61fa0221d0c7e7534e1ae688772a8ed4
-  data.tar.gz: 7173644a027051dfcd4911b0611995bac6686c9a78cd104836c5b3fbccf6d9dc
+  metadata.gz: 5450063206508cf94195d16dfe5be8db7609b4b959a9e2e36ca7cc102b90e04b
+  data.tar.gz: 6b10b7ddc67fdff01b31a78e486a4ea4f351277e7f851071406237300177d614
 SHA512:
-  metadata.gz: fb0400861a8946176b1ed8d63cc5504743696ec456c637b459c59489897fcce388e4defba72ecaf061e70362965792474037aef4bcdee48bb071d9f22002611e
-  data.tar.gz: f7b4f080b7b20b716950ff80ce5eff7bb1cc68c153bb65e2ef797044222dfcf52a5717e9a640817a946a54713ca164fb20ed7acbbc90bfec1effe57015c3d872
+  metadata.gz: 71a4c19d255e551e3a44aa1cdc508c5cb8dee3d66ae27c5b67a7fe2aaa31f958226e2a947bba773738274f717e5f635e412da1c471a03608657fc3deead4a961
+  data.tar.gz: 602c99609a6b65d3edbd29c8b8c26e9eed040830a115f207c0cebe139da80c6e43ccd2c1d583f4920fc0550c546747055f7da2fa73d858b9d20b4c7e37e674a9

data/README.md

@@ -1,27 +1,30 @@
 # Solid Queue
 
-Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
+Solid Queue is a database-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, recurring jobs, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
+In addition to regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, recurring jobs, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
 
-Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails's multi-threading.
+Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, and delays, and it's compatible with Ruby on Rails's multi-threading.
 
-## Table of contents
+## Table of Contents
 
 - [Installation](#installation)
+  - [Usage in development and other non-production environments](#usage-in-development-and-other-non-production-environments)
   - [Single database configuration](#single-database-configuration)
+  - [Dashboard UI Setup](#dashboard-ui-setup)
   - [Incremental adoption](#incremental-adoption)
   - [High performance requirements](#high-performance-requirements)
 - [Configuration](#configuration)
-  - [Workers, dispatchers and scheduler](#workers-dispatchers-and-scheduler)
+  - [Workers, dispatchers, and scheduler](#workers-dispatchers-and-scheduler)
   - [Queue order and priorities](#queue-order-and-priorities)
   - [Queues specification and performance](#queues-specification-and-performance)
-  - [Threads, processes and signals](#threads-processes-and-signals)
+  - [Threads, processes, and signals](#threads-processes-and-signals)
   - [Database configuration](#database-configuration)
   - [Other configuration settings](#other-configuration-settings)
 - [Lifecycle hooks](#lifecycle-hooks)
 - [Errors when enqueuing](#errors-when-enqueuing)
 - [Concurrency controls](#concurrency-controls)
+  - [Performance considerations](#performance-considerations)
 - [Failed jobs and retries](#failed-jobs-and-retries)
   - [Error reporting on jobs](#error-reporting-on-jobs)
 - [Puma plugin](#puma-plugin)
@@ -33,14 +36,16 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite,
 
 ## Installation
 
-Solid Queue is configured by default in new Rails 8 applications. But if you're running an earlier version, you can add it manually following these steps:
+Solid Queue is configured by default in new Rails 8 applications. If you're running an earlier version, you can add it manually following these steps:
 
 1. `bundle add solid_queue`
 2. `bin/rails solid_queue:install`
 
+(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.1.6.)
+
 This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue.
 
-Once you've done that, you will then have to add the configuration for the queue database in `config/database.yml`. If you're using SQLite, it'll look like this:
+Once you've done that, you will have to add the configuration for the queue database in `config/database.yml`. If you're using SQLite, it'll look like this:
 
 ```yaml
 production:
@@ -74,7 +79,7 @@ Now you're ready to start processing jobs by running `bin/jobs` on the server th
 
 For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
-**Note**: future changes to the schema will come in the form of regular migrations.
+**Note**: Future changes to the schema will come in the form of regular migrations.
 
 ### Usage in development and other non-production environments
 
@@ -84,7 +89,7 @@ For example, if you're using SQLite in development, update `database.yml` as fol
 
 ```diff
 development:
-  primary:
++ primary:
     <<: *default
     database: storage/development.sqlite3
 +  queue:
@@ -145,7 +150,7 @@ development:
 
 ### Single database configuration
 
-Running Solid Queue in a separate database is recommended, but it's also possible to use one single database for both the app and the queue. Just follow these steps:
+Running Solid Queue in a separate database is recommended, but it's also possible to use one single database for both the app and the queue. Follow these steps:
 
 1. Copy the contents of `db/queue_schema.rb` into a normal migration and delete `db/queue_schema.rb`
 2. Remove `config.solid_queue.connects_to` from `production.rb`
@@ -153,6 +158,10 @@ Running Solid Queue in a separate database is recommended, but it's also possibl
 
 You won't have multiple databases, so `database.yml` doesn't need to have primary and queue database.
 
+### Dashboard UI Setup
+
+For viewing information about your jobs via a UI, we recommend taking a look at [mission_control-jobs](https://github.com/rails/mission_control-jobs), a dashboard where, among other things, you can examine and retry/discard failed jobs.
+
 ### Incremental adoption
 
 If you're planning to adopt Solid Queue incrementally by switching one job at the time, you can do so by leaving the `config.active_job.queue_adapter` set to your old backend, and then set the `queue_adapter` directly in the jobs you're moving:
@@ -172,7 +181,7 @@ Solid Queue was designed for the highest throughput when used with MySQL 8+ or P
 
 ## Configuration
 
-### Workers, dispatchers and scheduler
+### Workers, dispatchers, and scheduler
 
 We have several types of actors in Solid Queue:
 
@@ -189,6 +198,8 @@ By default, Solid Queue will try to find your configuration under `config/queue.
 bin/jobs -c config/calendar.yml
 ```
 
+You can also skip all recurring tasks by setting the environment variable `SOLID_QUEUE_SKIP_RECURRING=true`. This is useful for environments like staging, review apps, or development where you don't want any recurring jobs to run. This is equivalent to using the `--skip-recurring` option with `bin/jobs`.
+
 This is what this configuration looks like:
 
 ```yml
@@ -310,7 +321,7 @@ and then remove the paused ones. Pausing in general should be something rare, us
 Do this:
 
 ```yml
-queues: background, backend
+queues: [ background, backend ]
 ```
 
 instead of this:
@@ -319,7 +330,7 @@ queues: back*
 ```
 
 
-### Threads, processes and signals
+### Threads, processes, and signals
 
 Workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling.
 
@@ -331,7 +342,7 @@ When receiving a `QUIT` signal, if workers still have jobs in-flight, these will
 
 If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. Jobs that were claimed by processes with an expired heartbeat will be marked as failed with a `SolidQueue::Processes::ProcessPrunedError`. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
 
-In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::Process::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.
+In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.
 
 
 ### Database configuration
@@ -355,14 +366,14 @@ There are several settings that control how Solid Queue works that you can set a
 
   **This is not used for errors raised within a job execution**. Errors happening in jobs are handled by Active Job's `retry_on` or `discard_on`, and ultimately will result in [failed jobs](#failed-jobs-and-retries). This is for errors happening within Solid Queue itself.
 
-- `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
+- `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
 - `shutdown_timeout`: time the supervisor will wait since it sent the `TERM` signal to its supervised processes before sending a `QUIT` version to them requesting immediate termination—defaults to 5 seconds.
 - `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to `true`.
 - `supervisor_pidfile`: path to a pidfile that the supervisor will create when booting to prevent running more than one supervisor in the same host, or in case you want to use it for a health check. It's `nil` by default.
 - `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to `true`.
-- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, which can be configured as [a recurring task](#recurring-tasks).
+- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true — defaults to 1 day. When installing Solid Queue, [a recurring job](#recurring-tasks) is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the `recurring.yml` configuration to change this as you see fit.
 - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
 
 
@@ -372,9 +383,11 @@ In Solid queue, you can hook into two different points in the supervisor's life:
 - `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
 - `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
 
-And into two different points in a worker's life:
-- `worker_start`: after the worker has finished booting and right before it starts the polling loop.
-- `worker_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
+And into two different points in the worker's, dispatcher's and scheduler's life:
+- `(worker|dispatcher|scheduler)_start`: after the worker/dispatcher/scheduler has finished booting and right before it starts the polling loop or loading the recurring schedule.
+- `(worker|dispatcher|scheduler)_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
+
+Each of these hooks has an instance of the supervisor/worker/dispatcher/scheduler yielded to the block so that you may read its configuration for logging or metrics reporting purposes.
 
 You can use the following methods with a block to do this:
 ```ruby
@@ -383,12 +396,30 @@ SolidQueue.on_stop
 
 SolidQueue.on_worker_start
 SolidQueue.on_worker_stop
+
+SolidQueue.on_dispatcher_start
+SolidQueue.on_dispatcher_stop
+
+SolidQueue.on_scheduler_start
+SolidQueue.on_scheduler_stop
 ```
 
 For example:
 ```ruby
-SolidQueue.on_start { start_metrics_server }
-SolidQueue.on_stop { stop_metrics_server }
+SolidQueue.on_start do |supervisor|
+  MyMetricsReporter.process_name = supervisor.name
+
+  start_metrics_server
+end
+
+SolidQueue.on_stop do |_supervisor|
+  stop_metrics_server
+end
+
+SolidQueue.on_worker_start do |worker|
+  MyMetricsReporter.process_name = worker.name
+  MyMetricsReporter.queues = worker.queues.join(',')
+end
 ```
 
 These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
@@ -402,11 +433,13 @@ In the case of recurring tasks, if such error is raised when enqueuing the job c
 
 ## Concurrency controls
 
-Solid Queue extends Active Job with concurrency controls, that allows you to limit how many jobs of a certain type or with certain arguments can run at the same time. When limited in this way, jobs will be blocked from running, and they'll stay blocked until another job finishes and unblocks them, or after the set expiry time (concurrency limit's _duration_) elapses. Jobs are never discarded or lost, only blocked.
+Solid Queue extends Active Job with concurrency controls, that allows you to limit how many jobs of a certain type or with certain arguments can run at the same time. When limited in this way, **by default, jobs will be blocked from running**, and they'll stay blocked until another job finishes and unblocks them, or after the set expiry time (concurrency limit's _duration_) elapses.
+
+**Alternatively, jobs can be configured to be discarded instead of blocked**. This means that if a job with certain arguments has already been enqueued, other jobs with the same characteristics (in the same concurrency _class_) won't be enqueued.
 
 ```ruby
 class MyJob < ApplicationJob
-  limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, **) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group
+  limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, **) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour
 
   # ...
 ```
@@ -414,10 +447,19 @@ class MyJob < ApplicationJob
 - `to` is `1` by default.
 - `duration` is set to `SolidQueue.default_concurrency_control_period` by default, which itself defaults to `3 minutes`, but that you can configure as well.
 - `group` is used to control the concurrency of different job classes together. It defaults to the job class name.
+- `on_conflict` controls behaviour when enqueuing a job that conflicts with the concurrency limits configured. It can be set to one of the following:
+  - (default) `:block`: the job is blocked and is dispatched when another job completes and unblocks it, or when the duration expires.
+  - `:discard`: the job is discarded. When you choose this option, bear in mind that if a job runs and fails to remove the concurrency lock (or _semaphore_, read below to know more about this), all jobs conflicting with it will be discarded up to the interval defined by `duration` has elapsed.
 
 When a job includes these controls, we'll ensure that, at most, the number of jobs (indicated as `to`) that yield the same `key` will be performed concurrently, and this guarantee will last for `duration` for each job enqueued. Note that there's no guarantee about _the order of execution_, only about jobs being performed at the same time (overlapping).
 
-The concurrency limits use the concept of semaphores when enqueuing, and work as follows: when a job is enqueued, we check if it specifies concurrency controls. If it does, we check the semaphore for the computed concurrency key. If the semaphore is open, we claim it and we set the job as _ready_. Ready means it can be picked up by workers for execution. When the job finishes executing (be it successfully or unsuccessfully, resulting in a failed execution), we signal the semaphore and try to unblock the next job with the same key, if any. Unblocking the next job doesn't mean running that job right away, but moving it from _blocked_ to _ready_. Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the `duration` as a failsafe. Jobs that have been blocked for more than duration are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the `duration` is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting.
+The concurrency limits use the concept of semaphores when enqueuing, and work as follows: when a job is enqueued, we check if it specifies concurrency controls. If it does, we check the semaphore for the computed concurrency key. If the semaphore is open, we claim it and we set the job as _ready_. Ready means it can be picked up by workers for execution. When the job finishes executing (be it successfully or unsuccessfully, resulting in a failed execution), we signal the semaphore and try to unblock the next job with the same key, if any. Unblocking the next job doesn't mean running that job right away, but moving it from _blocked_ to _ready_. If you're using the `discard` behaviour for `on_conflict`, jobs enqueued while the semaphore is closed will be discarded.
+
+Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the `duration` as a failsafe. Jobs that have been blocked for more than `duration` are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the `duration` is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting, or about the jobs that would get discarded while the semaphore is closed.
+
+It's important to note that after one or more candidate jobs are unblocked (either because a job finishes or because `duration` expires and a semaphore is released), the `duration` timer for the still blocked jobs is reset. This happens indirectly via the expiration time of the semaphore, which is updated.
+
+When using `discard` as the behaviour to handle conflicts, you might have jobs discarded for up to the `duration` interval if something happens and a running job fails to release the semaphore.
 
 
 For example:
@@ -450,12 +492,63 @@ class Bundle::RebundlePostingsJob < ApplicationJob
 
 In this case, if we have a `Box::MovePostingsByContactToDesignatedBoxJob` job enqueued for a contact record with id `123` and another `Bundle::RebundlePostingsJob` job enqueued simultaneously for a bundle record that references contact `123`, only one of them will be allowed to proceed. The other one will stay blocked until the first one finishes (or 15 minutes pass, whatever happens first).
 
-Note that the `duration` setting depends indirectly on the value for `concurrency_maintenance_interval` that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked. In general, you should set `duration` in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.
+Note that the `duration` setting depends indirectly on the value for `concurrency_maintenance_interval` that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked (at which point, only one job per concurrency key, at most, is unblocked). In general, you should set `duration` in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.
 
-Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
+Jobs are unblocked in order of priority but **queue order is not taken into account for unblocking jobs**. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
 
 Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for getting an open semaphore, and whenever they get it, they'll be run. It doesn't matter if they had already gotten an open semaphore in the past.
 
+### Scheduled jobs
+
+Jobs set to run in the future (via Active Job's `wait` or `wait_until` options) have concurrency limits enforced when they're due, not when they're scheduled. For example, consider this job:
+```ruby
+class DeliverAnnouncementToContactJob < ApplicationJob
+  limits_concurrency to: 1, key: ->(contact) { contact.account }, duration: 5.minutes
+
+  def perform(contact)
+    # ...
+```
+
+If several jobs are enqueued like this:
+
+```ruby
+DeliverAnnouncementToContactJob.set(wait: 10.minutes).perform_later(contact)
+DeliverAnnouncementToContactJob.set(wait: 10.minutes).perform_later(contact)
+DeliverAnnouncementToContactJob.set(wait: 30.minutes).perform_later(contact)
+```
+
+The 3 jobs will go into the scheduled queue and will wait there until they're due. Then, 10 minutes after, the first two jobs will be enqueued and the second one most likely will be blocked because the first one will be running first. Then, assuming the jobs are fast and finish in a few seconds, when the third job is due, it'll be enqueued normally.
+
+Normally scheduled jobs are enqueued in batches, but with concurrency controls, jobs need to be enqueued one by one. This has an impact on performance, similarly to the impact of concurrency controls in bulk enqueuing. Read below for more details. I'd generally advise against mixing concurrency controls with waiting/scheduling in the future.
+
+### Performance considerations
+
+Concurrency controls introduce significant overhead (blocked executions need to be created and promoted to ready, semaphores need to be created and updated) so you should consider carefully whether you need them. For throttling purposes, where you plan to have `limit` significantly larger than 1, I'd encourage relying on a limited number of workers per queue instead. For example:
+
+```ruby
+class ThrottledJob < ApplicationJob
+  queue_as :throttled
+```
+
+```yml
+production:
+  workers:
+    - queues: throttled
+      threads: 1
+      polling_interval: 1
+    - queues: default
+      threads: 5
+      polling_interval: 0.1
+      processes: 3
+```
+
+Or something similar to that depending on your setup. You can also assign a different queue to a job on the moment of enqueuing so you can decide whether to enqueue a job in the throttled queue or another queue depending on the arguments, or pass a block to `queue_as` as explained [here](https://guides.rubyonrails.org/active_job_basics.html#queues).
+
+
+In addition, mixing concurrency controls with **bulk enqueuing** (Active Job's `perform_all_later`) is not a good idea because concurrency controlled job needs to be enqueued one by one to ensure concurrency limits are respected, so you lose all the benefits of bulk enqueuing.
+
+When jobs that have concurrency controls and `on_conflict: :discard` are enqueued in bulk, the ones that fail to be enqueued and are discarded would have `successfully_enqueued` set to `false`. The total count of jobs enqueued returned by `perform_all_later` will exclude these jobs as expected.
+
 ## Failed jobs and retries
 
 Solid Queue doesn't include any automatic retry mechanism, it [relies on Active Job for this](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs). Jobs that fail will be kept in the system, and a _failed execution_ (a record in the `solid_queue_failed_executions` table) will be created for these. The job will stay there until manually discarded or re-enqueued. You can do this in a console as:
@@ -467,8 +560,6 @@ failed_execution.retry # This will re-enqueue the job as if it was enqueued for
 failed_execution.discard # This will delete the job from the system
 ```
 
-However, we recommend taking a look at [mission_control-jobs](https://github.com/rails/mission_control-jobs), a dashboard where, among other things, you can examine and retry/discard failed jobs.
-
 ### Error reporting on jobs
 
 Some error tracking services that integrate with Rails, such as Sentry or Rollbar, hook into [Active Job](https://guides.rubyonrails.org/active_job_basics.html#exceptions) and automatically report not handled errors that happen during job execution. However, if your error tracking system doesn't, or if you need some custom reporting, you can hook into Active Job yourself. A possible way of doing this would be:
@@ -510,6 +601,7 @@ plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]
 ```
 that you set in production only. This is what Rails 8's default Puma config looks like. Otherwise, if you're using Puma in development but not Solid Queue, starting Puma would start also Solid Queue supervisor and it'll most likely fail because it won't be properly configured.
 
+**Note**: phased restarts are not supported currently because the plugin requires [app preloading](https://github.com/puma/puma?tab=readme-ov-file#cluster-mode) to work.
 
 ## Jobs and transactional integrity
 :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed and vice versa, and ensuring that your job won't be enqueued until the transaction within which you're enqueuing it is committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you. Because this can be quite tricky and many people shouldn't need to worry about it, by default Solid Queue is configured in a different database as the main app.
@@ -524,7 +616,7 @@ end
 
 Using this option, you can also use Solid Queue in the same database as your app but not rely on transactional integrity.
 
-If you don't set this option but still want to make sure you're not inadvertently on transactional integrity, you can make sure that:
+If you don't set this option but still want to make sure you're not inadvertently relying on transactional integrity, you can make sure that:
 - Your jobs relying on specific data are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
 - Or, you configure a different database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
 
@@ -548,6 +640,8 @@ Solid Queue supports defining recurring tasks that run at specific times in the
 bin/jobs --recurring_schedule_file=config/schedule.yml
 ```
 
+You can completely disable recurring tasks by setting the environment variable `SOLID_QUEUE_SKIP_RECURRING=true` or by using the `--skip-recurring` option with `bin/jobs`.
+
 The configuration itself looks like this:
 
 ```yml

data/app/models/solid_queue/blocked_execution.rb

@@ -12,7 +12,7 @@ module SolidQueue
     class << self
       def unblock(limit)
         SolidQueue.instrument(:release_many_blocked, limit: limit) do |payload|
-          expired.distinct.limit(limit).pluck(:concurrency_key).then do |concurrency_keys|
+          expired.order(:concurrency_key).distinct.limit(limit).pluck(:concurrency_key).then do |concurrency_keys|
             payload[:size] = release_many releasable(concurrency_keys)
           end
         end

data/app/models/solid_queue/claimed_execution.rb

@@ -37,9 +37,14 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     end
 
     def fail_all_with(error)
-      SolidQueue.instrument(:fail_many_claimed) do |payload|
-        includes(:job).tap do |executions|
-          executions.each { |execution| execution.failed_with(error) }
+      includes(:job).tap do |executions|
+        return if executions.empty?
+
+        SolidQueue.instrument(:fail_many_claimed) do |payload|
+          executions.each do |execution|
+            execution.failed_with(error)
+            execution.unblock_next_job
+          end
 
           payload[:process_ids] = executions.map(&:process_id).uniq
           payload[:job_ids] = executions.map(&:job_id).uniq
@@ -67,7 +72,7 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
       raise result.error
     end
   ensure
-    job.unblock_next_blocked_job
+    unblock_next_job
   end
 
   def release
@@ -90,9 +95,13 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     end
   end
 
+  def unblock_next_job
+    job.unblock_next_blocked_job
+  end
+
   private
     def execute
-      ActiveJob::Base.execute(job.arguments)
+      ActiveJob::Base.execute(job.arguments.merge("provider_job_id" => job.id))
       Result.new(true, nil)
     rescue Exception => e
       Result.new(false, e)

data/app/models/solid_queue/failed_execution.rb

@@ -58,8 +58,11 @@ module SolidQueue
       end
 
       def determine_backtrace_size_limit
-        column = self.class.connection.schema_cache.columns_hash(self.class.table_name)["error"]
-        if column.limit.present?
+        column = self.class.connection_pool.with_connection do |connection|
+          connection.schema_cache.columns_hash(self.class.table_name)["error"]
+        end
+
+        if column && column.limit.present?
           column.limit - exception_class_name.bytesize - exception_message.bytesize - JSON_OVERHEAD
         end
       end

data/app/models/solid_queue/job/concurrency_controls.rb

@@ -34,6 +34,10 @@ module SolidQueue
       end
 
       private
+        def concurrency_on_conflict
+          job_class.concurrency_on_conflict.to_s.inquiry
+        end
+
         def acquire_concurrency_lock
           return true unless concurrency_limited?
 
@@ -46,6 +50,14 @@ module SolidQueue
           Semaphore.signal(self)
         end
 
+        def handle_concurrency_conflict
+          if concurrency_on_conflict.discard?
+            destroy
+          else
+            block
+          end
+        end
+
         def block
           BlockedExecution.create_or_find_by!(job_id: id)
         end

data/app/models/solid_queue/job/executable.rb

@@ -67,7 +67,7 @@ module SolidQueue
       def dispatch
         if acquire_concurrency_lock then ready
         else
-          block
+          handle_concurrency_conflict
         end
       end
 

data/app/models/solid_queue/job.rb

@@ -10,6 +10,7 @@ module SolidQueue
 
     class << self
       def enqueue_all(active_jobs)
+        active_jobs.each { |job| job.scheduled_at ||= Time.current }
         active_jobs_by_job_id = active_jobs.index_by(&:job_id)
 
         transaction do
@@ -29,7 +30,8 @@ module SolidQueue
         active_job.scheduled_at = scheduled_at
 
         create_from_active_job(active_job).tap do |enqueued_job|
-          active_job.provider_job_id = enqueued_job.id
+          active_job.provider_job_id = enqueued_job.id if enqueued_job.persisted?
+          active_job.successfully_enqueued = enqueued_job.persisted?
         end
       end
 
@@ -49,7 +51,7 @@ module SolidQueue
         def create_all_from_active_jobs(active_jobs)
           job_rows = active_jobs.map { |job| attributes_from_active_job(job) }
           insert_all(job_rows)
-          where(active_job_id: active_jobs.map(&:job_id))
+          where(active_job_id: active_jobs.map(&:job_id)).order(id: :asc)
         end
 
         def attributes_from_active_job(active_job)

data/app/models/solid_queue/record.rb

@@ -6,11 +6,19 @@ module SolidQueue
 
     connects_to(**SolidQueue.connects_to) if SolidQueue.connects_to
 
-    def self.non_blocking_lock
-      if SolidQueue.use_skip_locked
-        lock(Arel.sql("FOR UPDATE SKIP LOCKED"))
-      else
-        lock
+    class << self
+      def non_blocking_lock
+        if SolidQueue.use_skip_locked
+          lock(Arel.sql("FOR UPDATE SKIP LOCKED"))
+        else
+          lock
+        end
+      end
+
+      def supports_insert_conflict_target?
+        connection_pool.with_connection do |connection|
+          connection.supports_insert_conflict_target?
+        end
       end
     end
   end

data/app/models/solid_queue/recurring_execution.rb

@@ -8,7 +8,7 @@ module SolidQueue
 
     class << self
       def create_or_insert!(**attributes)
-        if connection.supports_insert_conflict_target?
+        if supports_insert_conflict_target?
           # PostgreSQL fails and aborts the current transaction when it hits a duplicate key conflict
           # during two concurrent INSERTs for the same value of an unique index. We need to explicitly
           # indicate unique_by to ignore duplicate rows by this value when inserting

data/app/models/solid_queue/recurring_task.rb

@@ -36,7 +36,7 @@ module SolidQueue
       end
 
       def create_or_update_all(tasks)
-        if connection.supports_insert_conflict_target?
+        if supports_insert_conflict_target?
           # PostgreSQL fails and aborts the current transaction when it hits a duplicate key conflict
           # during two concurrent INSERTs for the same value of an unique index. We need to explicitly
           # indicate unique_by to ignore duplicate rows by this value when inserting
@@ -48,7 +48,7 @@ module SolidQueue
     end
 
     def delay_from_now
-      [ (next_time - Time.current).to_f, 0 ].max
+      [ (next_time - Time.current).to_f, 0.1 ].max
     end
 
     def next_time
@@ -130,7 +130,6 @@ module SolidQueue
             active_job.run_callbacks(:enqueue) do
               Job.enqueue(active_job)
             end
-            active_job.successfully_enqueued = true
           end
         end
       end

data/app/models/solid_queue/scheduled_execution.rb

@@ -14,7 +14,7 @@ module SolidQueue
       def dispatch_next_batch(batch_size)
         transaction do
           job_ids = next_batch(batch_size).non_blocking_lock.pluck(:job_id)
-          if job_ids.empty? then []
+          if job_ids.empty? then 0
           else
             SolidQueue.instrument(:dispatch_scheduled, batch_size: batch_size) do |payload|
               payload[:size] = dispatch_jobs(job_ids)

data/app/models/solid_queue/semaphore.rb

@@ -20,7 +20,7 @@ module SolidQueue
 
       # Requires a unique index on key
       def create_unique_by(attributes)
-        if connection.supports_insert_conflict_target?
+        if supports_insert_conflict_target?
           insert({ **attributes }, unique_by: :key).any?
         else
           create!(**attributes)

data/lib/active_job/concurrency_controls.rb

@@ -5,6 +5,7 @@ module ActiveJob
     extend ActiveSupport::Concern
 
     DEFAULT_CONCURRENCY_GROUP = ->(*) { self.class.name }
+    CONCURRENCY_ON_CONFLICT_BEHAVIOUR = %i[ block discard ]
 
     included do
       class_attribute :concurrency_key, instance_accessor: false
@@ -12,14 +13,16 @@ module ActiveJob
 
       class_attribute :concurrency_limit
       class_attribute :concurrency_duration, default: SolidQueue.default_concurrency_control_period
+      class_attribute :concurrency_on_conflict, default: :block
     end
 
     class_methods do
-      def limits_concurrency(key:, to: 1, group: DEFAULT_CONCURRENCY_GROUP, duration: SolidQueue.default_concurrency_control_period)
+      def limits_concurrency(key:, to: 1, group: DEFAULT_CONCURRENCY_GROUP, duration: SolidQueue.default_concurrency_control_period, on_conflict: :block)
         self.concurrency_key = key
         self.concurrency_limit = to
         self.concurrency_group = group
         self.concurrency_duration = duration
+        self.concurrency_on_conflict = on_conflict.presence_in(CONCURRENCY_ON_CONFLICT_BEHAVIOUR) || :block
       end
     end
 

data/lib/active_job/queue_adapters/solid_queue_adapter.rb

@@ -7,7 +7,10 @@ module ActiveJob
     # To use it set the queue_adapter config to +:solid_queue+.
     #
     #   Rails.application.config.active_job.queue_adapter = :solid_queue
-    class SolidQueueAdapter
+    class SolidQueueAdapter < (Rails::VERSION::MAJOR == 7 && Rails::VERSION::MINOR == 1 ? Object : AbstractAdapter)
+      class_attribute :stopping, default: false, instance_writer: false
+      SolidQueue.on_worker_stop { self.stopping = true }
+
       def enqueue_after_transaction_commit?
         true
       end

data/lib/generators/solid_queue/install/templates/config/recurring.yml

@@ -1,10 +1,15 @@
-# production:
+# examples:
 #   periodic_cleanup:
 #     class: CleanSoftDeletedRecordsJob
 #     queue: background
 #     args: [ 1000, { batch_size: 500 } ]
 #     schedule: every hour
-#   periodic_command:
+#   periodic_cleanup_with_command:
 #     command: "SoftDeletedRecord.due.delete_all"
 #     priority: 2
 #     schedule: at 5am every day
+
+production:
+  clear_solid_queue_finished_jobs:
+    command: "SolidQueue::Job.clear_finished_in_batches(sleep_between_batches: 0.3)"
+    schedule: every hour at minute 12

data/lib/puma/plugin/solid_queue.rb

@@ -11,15 +11,27 @@ Puma::Plugin.create do
       monitor_solid_queue
     end
 
-    launcher.events.on_booted do
-      @solid_queue_pid = fork do
-        Thread.new { monitor_puma }
-        SolidQueue::Supervisor.start
+    if Gem::Version.new(Puma::Const::VERSION) < Gem::Version.new("7")
+      launcher.events.on_booted do
+        @solid_queue_pid = fork do
+          Thread.new { monitor_puma }
+          SolidQueue::Supervisor.start
+        end
+      end
+
+      launcher.events.on_stopped { stop_solid_queue }
+      launcher.events.on_restart { stop_solid_queue }
+    else
+      launcher.events.after_booted do
+        @solid_queue_pid = fork do
+          Thread.new { monitor_puma }
+          SolidQueue::Supervisor.start
+        end
       end
-    end
 
-    launcher.events.on_stopped { stop_solid_queue }
-    launcher.events.on_restart { stop_solid_queue }
+      launcher.events.after_stopped { stop_solid_queue }
+      launcher.events.before_restart { stop_solid_queue }
+    end
   end
 
   private

data/lib/solid_queue/cli.rb

@@ -12,8 +12,9 @@ module SolidQueue
       desc: "Path to recurring schedule definition (default: #{Configuration::DEFAULT_RECURRING_SCHEDULE_FILE_PATH}).",
       banner: "SOLID_QUEUE_RECURRING_SCHEDULE"
 
-    class_option :skip_recurring, type: :boolean, default: false,
-      desc: "Whether to skip recurring tasks scheduling"
+    class_option :skip_recurring, type: :boolean,
+      desc: "Whether to skip recurring tasks scheduling",
+      banner: "SOLID_QUEUE_SKIP_RECURRING"
 
     def self.exit_on_failure?
       true

data/lib/solid_queue/configuration.rb

@@ -88,7 +88,7 @@ module SolidQueue
           recurring_schedule_file: Rails.root.join(ENV["SOLID_QUEUE_RECURRING_SCHEDULE"] || DEFAULT_RECURRING_SCHEDULE_FILE_PATH),
           only_work: false,
           only_dispatch: false,
-          skip_recurring: false
+          skip_recurring: ActiveModel::Type::Boolean.new.cast(ENV["SOLID_QUEUE_SKIP_RECURRING"])
         }
       end
 
@@ -141,7 +141,7 @@ module SolidQueue
 
       def recurring_tasks
         @recurring_tasks ||= recurring_tasks_config.map do |id, options|
-          RecurringTask.from_configuration(id, **options) if options.has_key?(:schedule)
+          RecurringTask.from_configuration(id, **options) if options&.has_key?(:schedule)
         end.compact
       end
 
@@ -153,7 +153,9 @@ module SolidQueue
       end
 
       def recurring_tasks_config
-        @recurring_tasks_config ||= config_from options[:recurring_schedule_file]
+        @recurring_tasks_config ||= begin
+          config_from options[:recurring_schedule_file]
+        end
       end
 
 
@@ -186,6 +188,7 @@ module SolidQueue
         if file.exist?
           ActiveSupport::ConfigurationFile.parse(file).deep_symbolize_keys
         else
+          puts "[solid_queue] WARNING: Provided configuration file '#{file}' does not exist. Falling back to default configuration."
           {}
         end
       end

data/lib/solid_queue/dispatcher.rb

@@ -2,10 +2,14 @@
 
 module SolidQueue
   class Dispatcher < Processes::Poller
-    attr_accessor :batch_size, :concurrency_maintenance
+    include LifecycleHooks
+    attr_reader :batch_size
 
+    after_boot :run_start_hooks
     after_boot :start_concurrency_maintenance
     before_shutdown :stop_concurrency_maintenance
+    before_shutdown :run_stop_hooks
+    after_shutdown :run_exit_hooks
 
     def initialize(**options)
       options = options.dup.with_defaults(SolidQueue::Configuration::DISPATCHER_DEFAULTS)
@@ -22,10 +26,12 @@ module SolidQueue
     end
 
     private
+      attr_reader :concurrency_maintenance
+
       def poll
         batch = dispatch_next_batch
 
-        batch.size.zero? ? polling_interval : 0.seconds
+        batch.zero? ? polling_interval : 0.seconds
       end
 
       def dispatch_next_batch
@@ -38,20 +44,12 @@ module SolidQueue
         concurrency_maintenance&.start
       end
 
-      def schedule_recurring_tasks
-        recurring_schedule.schedule_tasks
-      end
-
       def stop_concurrency_maintenance
         concurrency_maintenance&.stop
       end
 
-      def unschedule_recurring_tasks
-        recurring_schedule.unschedule_tasks
-      end
-
       def all_work_completed?
-        SolidQueue::ScheduledExecution.none? && recurring_schedule.empty?
+        SolidQueue::ScheduledExecution.none?
       end
 
       def set_procline

data/lib/solid_queue/lifecycle_hooks.rb

@@ -5,7 +5,7 @@ module SolidQueue
     extend ActiveSupport::Concern
 
     included do
-      mattr_reader :lifecycle_hooks, default: { start: [], stop: [] }
+      mattr_reader :lifecycle_hooks, default: { start: [], stop: [], exit: [] }
     end
 
     class_methods do
@@ -17,7 +17,12 @@ module SolidQueue
         self.lifecycle_hooks[:stop] << block
       end
 
+      def on_exit(&block)
+        self.lifecycle_hooks[:exit] << block
+      end
+
       def clear_hooks
+        self.lifecycle_hooks[:exit] = []
         self.lifecycle_hooks[:start] = []
         self.lifecycle_hooks[:stop] = []
       end
@@ -32,9 +37,13 @@ module SolidQueue
         run_hooks_for :stop
       end
 
+      def run_exit_hooks
+        run_hooks_for :exit
+      end
+
       def run_hooks_for(event)
         self.class.lifecycle_hooks.fetch(event, []).each do |block|
-          block.call
+            block.call(self)
         rescue Exception => exception
           handle_thread_error(exception)
         end

data/lib/solid_queue/pool.rb

@@ -18,20 +18,16 @@ module SolidQueue
     def post(execution)
       available_threads.decrement
 
-      future = Concurrent::Future.new(args: [ execution ], executor: executor) do |thread_execution|
+      Concurrent::Promises.future_on(executor, execution) do |thread_execution|
         wrap_in_app_executor do
           thread_execution.perform
         ensure
           available_threads.increment
           mutex.synchronize { on_idle.try(:call) if idle? }
         end
+      end.on_rejection! do |e|
+        handle_thread_error(e)
       end
-
-      future.add_observer do |_, _, error|
-        handle_thread_error(error) if error
-      end
-
-      future.execute
     end
 
     def idle_threads

data/lib/solid_queue/processes/base.rb

@@ -4,7 +4,8 @@ module SolidQueue
   module Processes
     class Base
       include Callbacks # Defines callbacks needed by other concerns
-      include AppExecutor, Registrable, Interruptible, Procline
+      include AppExecutor, Registrable, Procline
+      prepend Interruptible
 
       attr_reader :name
 

data/lib/solid_queue/processes/interruptible.rb

@@ -2,32 +2,39 @@
 
 module SolidQueue::Processes
   module Interruptible
+    def initialize(...)
+      super
+      @self_pipe = create_self_pipe
+    end
+
     def wake_up
       interrupt
     end
 
     private
+      SELF_PIPE_BLOCK_SIZE = 11
+
+      attr_reader :self_pipe
 
       def interrupt
-        queue << true
+        self_pipe[:writer].write_nonblock(".")
+      rescue Errno::EAGAIN, Errno::EINTR
+        # Ignore writes that would block and retry
+        # if another signal arrived while writing
+        retry
       end
 
-      # Sleeps for 'time'.  Can be interrupted asynchronously and return early via wake_up.
-      # @param time [Numeric] the time to sleep. 0 returns immediately.
-      # @return [true, nil]
-      # * returns `true` if an interrupt was requested via #wake_up between the
-      #   last call to `interruptible_sleep` and now, resulting in an early return.
-      # * returns `nil` if it slept the full `time` and was not interrupted.
       def interruptible_sleep(time)
-        # Invoking this from the main thread may result in significant slowdown.
-        # Utilizing asynchronous execution (Futures) addresses this performance issue.
-        Concurrent::Promises.future(time) do |timeout|
-          queue.pop(timeout:).tap { queue.clear }
-        end.value
+        if time > 0 && self_pipe[:reader].wait_readable(time)
+          loop { self_pipe[:reader].read_nonblock(SELF_PIPE_BLOCK_SIZE) }
+        end
+      rescue Errno::EAGAIN, Errno::EINTR, IO::EWOULDBLOCKWaitReadable
       end
 
-      def queue
-        @queue ||= Queue.new
+      # Self-pipe for signal-handling (http://cr.yp.to/docs/selfpipe.html)
+      def create_self_pipe
+        reader, writer = IO.pipe
+        { reader: reader, writer: writer }
       end
   end
 end

data/lib/solid_queue/processes/process_pruned_error.rb

@@ -4,7 +4,7 @@ module SolidQueue
   module Processes
     class ProcessPrunedError < RuntimeError
       def initialize(last_heartbeat_at)
-        super("Process was found dead and pruned (last heartbeat at: #{last_heartbeat_at}")
+        super("Process was found dead and pruned (last heartbeat at: #{last_heartbeat_at})")
       end
     end
   end

data/lib/solid_queue/processes/registrable.rb

@@ -7,8 +7,7 @@ module SolidQueue::Processes
     included do
       after_boot :register, :launch_heartbeat
 
-      before_shutdown :stop_heartbeat
-      after_shutdown :deregister
+      after_shutdown :stop_heartbeat, :deregister
     end
 
     def process_id
@@ -19,17 +18,19 @@ module SolidQueue::Processes
       attr_accessor :process
 
       def register
-        @process = SolidQueue::Process.register \
-          kind: kind,
-          name: name,
-          pid: pid,
-          hostname: hostname,
-          supervisor: try(:supervisor),
-          metadata: metadata.compact
+        wrap_in_app_executor do
+          @process = SolidQueue::Process.register \
+            kind: kind,
+            name: name,
+            pid: pid,
+            hostname: hostname,
+            supervisor: try(:supervisor),
+            metadata: metadata.compact
+        end
       end
 
       def deregister
-        process&.deregister
+        wrap_in_app_executor { process&.deregister }
       end
 
       def registered?

data/lib/solid_queue/scheduler/recurring_schedule.rb

@@ -46,7 +46,7 @@ module SolidQueue
       end
 
       def reload_tasks
-        @configured_tasks = SolidQueue::RecurringTask.where(key: task_keys)
+        @configured_tasks = SolidQueue::RecurringTask.where(key: task_keys).to_a
       end
 
       def schedule(task)

data/lib/solid_queue/scheduler.rb

@@ -3,11 +3,15 @@
 module SolidQueue
   class Scheduler < Processes::Base
     include Processes::Runnable
+    include LifecycleHooks
 
-    attr_accessor :recurring_schedule
+    attr_reader :recurring_schedule
 
+    after_boot :run_start_hooks
     after_boot :schedule_recurring_tasks
     before_shutdown :unschedule_recurring_tasks
+    before_shutdown :run_stop_hooks
+    after_shutdown :run_exit_hooks
 
     def initialize(recurring_tasks:, **options)
       @recurring_schedule = RecurringSchedule.new(recurring_tasks)

data/lib/solid_queue/supervisor.rb

@@ -5,6 +5,8 @@ module SolidQueue
     include LifecycleHooks
     include Maintenance, Signals, Pidfiled
 
+    after_shutdown :run_exit_hooks
+
     class << self
       def start(**options)
         SolidQueue.supervisor = true
@@ -170,10 +172,15 @@ module SolidQueue
         end
       end
 
+      # When a supervised fork crashes or exits we need to mark all the
+      # executions it had claimed as failed so that they can be retried
+      # by some other worker.
       def handle_claimed_jobs_by(terminated_fork, status)
-        if registered_process = process.supervisees.find_by(name: terminated_fork.name)
-          error = Processes::ProcessExitError.new(status)
-          registered_process.fail_all_claimed_executions_with(error)
+        wrap_in_app_executor do
+          if registered_process = SolidQueue::Process.find_by(name: terminated_fork.name)
+            error = Processes::ProcessExitError.new(status)
+            registered_process.fail_all_claimed_executions_with(error)
+          end
         end
       end
 

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "1.1.2"
+  VERSION = "1.2.4"
 end

data/lib/solid_queue/worker.rb

@@ -6,14 +6,16 @@ module SolidQueue
 
     after_boot :run_start_hooks
     before_shutdown :run_stop_hooks
+    after_shutdown :run_exit_hooks
 
-
-    attr_accessor :queues, :pool
+    attr_reader :queues, :pool
 
     def initialize(**options)
       options = options.dup.with_defaults(SolidQueue::Configuration::WORKER_DEFAULTS)
 
-      @queues = Array(options[:queues])
+      # Ensure that the queues array is deep frozen to prevent accidental modification
+      @queues = Array(options[:queues]).map(&:freeze).freeze
+
       @pool = Pool.new(options[:threads], on_idle: -> { wake_up })
 
       super(**options)

data/lib/solid_queue.rb

@@ -41,14 +41,20 @@ module SolidQueue
   mattr_accessor :clear_finished_jobs_after, default: 1.day
   mattr_accessor :default_concurrency_control_period, default: 3.minutes
 
-  delegate :on_start, :on_stop, to: Supervisor
+  delegate :on_start, :on_stop, :on_exit, to: Supervisor
 
-  def on_worker_start(...)
-    Worker.on_start(...)
-  end
+  [ Dispatcher, Scheduler, Worker ].each do |process|
+    define_singleton_method(:"on_#{process.name.demodulize.downcase}_start") do |&block|
+      process.on_start(&block)
+    end
+
+    define_singleton_method(:"on_#{process.name.demodulize.downcase}_stop") do |&block|
+      process.on_stop(&block)
+    end
 
-  def on_worker_stop(...)
-    Worker.on_stop(...)
+    define_singleton_method(:"on_#{process.name.demodulize.downcase}_exit") do |&block|
+      process.on_exit(&block)
+    end
   end
 
   def supervisor?

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 1.1.2
+  version: 1.2.4
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-12-27 00:00:00.000000000 Z
+date: 2025-10-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -72,28 +72,42 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.11.0
+        version: '1.11'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.11.0
+        version: '1.11'
 - !ruby/object:Gem::Dependency
   name: thor
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.3.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.3.1
+- !ruby/object:Gem::Dependency
+  name: appraisal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -126,16 +140,16 @@ dependencies:
   name: puma
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '7.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '7.0'
 - !ruby/object:Gem::Dependency
   name: mysql2
   requirement: !ruby/object:Gem::Requirement
@@ -220,6 +234,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: zeitwerk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.6.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.6.0
 description: Database-backed Active Job backend.
 email:
 - rosa@37signals.com
@@ -229,7 +257,6 @@ extra_rdoc_files: []
 files:
 - MIT-LICENSE
 - README.md
-- Rakefile
 - UPGRADING.md
 - app/jobs/solid_queue/recurring_job.rb
 - app/models/solid_queue/blocked_execution.rb
@@ -307,15 +334,8 @@ metadata:
   homepage_uri: https://github.com/rails/solid_queue
   source_code_uri: https://github.com/rails/solid_queue
 post_install_message: |
-  Upgrading to Solid Queue 0.9.0? There are some breaking changes about how recurring tasks are configured.
-
-  Upgrading to Solid Queue 0.8.0 from < 0.6.0? You need to upgrade to 0.6.0 first.
-
-  Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
-  configuration and new migrations.
-
-  --> Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
-  for upgrade instructions.
+  Upgrading from Solid Queue < 1.0? Check details on breaking changes and upgrade instructions
+  --> https://github.com/rails/solid_queue/blob/main/UPGRADING.md
 rdoc_options: []
 require_paths:
 - lib
@@ -323,7 +343,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '3.1'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="

data/Rakefile

@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-
-require "bundler/setup"
-
-APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
-load "rails/tasks/engine.rake"
-
-load "rails/tasks/statistics.rake"
-
-require "bundler/gem_tasks"
-
-def databases
-  %w[ mysql postgres sqlite ]
-end
-
-task :test do
-  databases.each do |database|
-    sh("TARGET_DB=#{database} bin/setup")
-    sh("TARGET_DB=#{database} bin/rails test")
-  end
-end