solid_queue 0.6.1 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c2734f6cbcc795345207dc0cf221bd2744f2f98449da8b0edfc8175f3bbd7869
-  data.tar.gz: c4b4c2eca7dcb93e86a2f69d220ba1b9f8817c1d52a7cd1ff137ed5ec84e50f8
+  metadata.gz: a7234dc4430998648196bf2d3905b66bb85e265c2393121a7c5343fed36b1996
+  data.tar.gz: 216a0918e29194e6d11fe4bf365183228127f8390658a11ace329523ad41468c
 SHA512:
-  metadata.gz: 9d09b58e43c4bc19ad2ab89d10072b36b53a449e9602f5c5a8fc7b637e91d8472eb70f417148657e6b146e5cd1034025afdbe6d39fa2b9fc82ce6bba3997e57f
-  data.tar.gz: 27168fd2216fbca4e9b19677a7885b23663012d8aed7bf54c25be8335bd528a733b55790e662a30815b9ba24d824390372eff24ed0b9f72947b6ff4feeec17c5
+  metadata.gz: a080aedf20f39940d8c25e8a14628811801b8fd4832fbcb926beecd0d1ce4c694ec8d80b608656f9de80cc8cc69b525f62d4fe7bc7258408aa6d6af62292d4fd
+  data.tar.gz: 66d24c1bb1c7cb1b9afbcf8a0b667df624dd3f47cea92887fbcd69d93b1832af3c44afb570d1cfb459c37e2b3c163900e0dd2febb067ee9179b4c49514e4b359

data/README.md

@@ -2,7 +2,7 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, and some way of specifying unique jobs are coming very soon._
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
@@ -31,9 +31,9 @@ $ bin/rails generate solid_queue:install
 
 This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
 
-Alternatively, you can add only the migration to your app:
+Alternatively, you can skip setting the Active Job's adapter with:
 ```bash
-$ bin/rails solid_queue:install:migrations
+$ bin/rails generate solid_queue:install --skip_adapter
 ```
 
 And set Solid Queue as your Active Job's queue backend manually, in your environment config:
@@ -42,7 +42,7 @@ And set Solid Queue as your Active Job's queue backend manually, in your environ
 config.active_job.queue_adapter = :solid_queue
 ```
 
-Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+Or you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
 
 ```ruby
 # app/jobs/my_job.rb
@@ -59,14 +59,14 @@ Finally, you need to run the migrations:
 $ bin/rails db:migrate
 ```
 
-After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them. You can use the provided binstub:`
 ```bash
-$ bundle exec rake solid_queue:start
+$ bin/jobs
 ```
 
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
-For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bundle exec rake solid_queue:start` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
+For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
 ## Requirements
 Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
@@ -80,7 +80,7 @@ We have three types of actors in Solid Queue:
 - _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They're also in charge of managing [recurring tasks](#recurring-tasks), dispatching jobs to process them according to their schedule. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
 - The _supervisor_ runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.
 
-By default, Solid Queue runs in `fork` mode. This means the supervisor will fork a separate process for each supervised worker/dispatcher. There's also an `async` mode where each worker and dispatcher will be run as a thread of the supervisor process. This can be used with [the provided Puma plugin](#puma-plugin)
+Solid Queue's supervisor will fork a separate process for each supervised worker/dispatcher.
 
 By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
 
@@ -131,7 +131,7 @@ Here's an overview of the different options:
 
   Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
 - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
-- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. **Note**: this option will be ignored if [running in `async` mode](#running-as-a-fork-or-asynchronously).
+- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 - `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
 
@@ -194,13 +194,13 @@ development:
     # ...
 ```
 
-Install migrations and specify the dedicated database name with the `DATABASE` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
+Install migrations and specify the dedicated database name with the `--database` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
 
 ```bash
-$ bin/rails solid_queue:install:migrations DATABASE=solid_queue
+$ bin/rails g solid_queue:install --database solid_queue
 ```
 
-Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`), the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
+Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`) without a `--database` option, the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
 
 Finally, run the migrations:
 
@@ -208,17 +208,47 @@ Finally, run the migrations:
 $ bin/rails db:migrate
 ```
 
+## Lifecycle hooks
+
+In Solid queue, you can hook into two different points in the supervisor's life:
+- `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
+- `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
+
+And into two different points in a worker's life:
+- `worker_start`: after the worker has finished booting and right before it starts the polling loop.
+- `worker_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
+
+You can use the following methods with a block to do this:
+```ruby
+SolidQueue.on_start
+SolidQueue.on_stop
+
+SolidQueue.on_worker_start
+SolidQueue.on_worker_stop
+```
+
+For example:
+```ruby
+SolidQueue.on_start { start_metrics_server }
+SolidQueue.on_stop { stop_metrics_server }
+```
+
+These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
+
+
 ### Other configuration settings
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
 
 There are several settings that control how Solid Queue works that you can set as well:
 - `logger`: the logger you want Solid Queue to use. Defaults to the app logger.
 - `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap asynchronous operations, defaults to the app executor
-- `on_thread_error`: custom lambda/Proc to call when there's an error within a thread that takes the exception raised as argument. Defaults to
+- `on_thread_error`: custom lambda/Proc to call when there's an error within a Solid Queue thread that takes the exception raised as argument. Defaults to
 
   ```ruby
   -> (exception) { Rails.error.report(exception, handled: false) }
   ```
+  **This is not used for errors raised within a job execution**. Errors happening in jobs are handled by Active Job's `retry_on` or `discard_on`, and ultimately will result in [failed jobs](#failed-jobs-and-retries). This is for errors happening within Solid Queue itself.
+
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
@@ -283,6 +313,8 @@ In this case, if we have a `Box::MovePostingsByContactToDesignatedBoxJob` job en
 
 Note that the `duration` setting depends indirectly on the value for `concurrency_maintenance_interval` that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked. In general, you should set `duration` in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.
 
+Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
+
 Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for gaining the lock, and whenever they get it, they'll be run. It doesn't matter if they had gained the lock already in the past.
 
 ## Failed jobs and retries
@@ -305,18 +337,6 @@ plugin :solid_queue
 ```
 to your `puma.rb` configuration.
 
-### Running as a fork or asynchronously
-
-By default, the Puma plugin will fork additional processes for each worker and dispatcher so that they run in different processes. This provides the best isolation and performance, but can have additional memory usage.
-
-Alternatively, workers and dispatchers can be run within the same Puma process(s). To do so just configure the plugin as:
-
-```ruby
-plugin :solid_queue
-solid_queue_mode :async
-```
-
-Note that in this case, the `processes` configuration option will be ignored.
 
 ## Jobs and transactional integrity
 :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.

data/UPGRADING.md

@@ -0,0 +1,102 @@
+# Upgrading to version 0.7.x
+
+This version removed the new async mode introduced in version 0.4.0 and introduced a new binstub that can be used to start Solid Queue's supervisor. It includes also a minor migration.
+
+To install both the binstub `bin/jobs` and the migration, you can just run
+```
+bin/rails generate solid_queue:install
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails generate solid_queue:install --database <the_name_of_your_solid_queue_db>
+```
+
+
+# Upgrading to version 0.6.x
+
+## New migration in 3 steps
+This version adds two new migrations to modify the `solid_queue_processes` table. The goal of that migration is to add a new column that needs to be `NOT NULL`. This needs to be done with two migrations and the following steps to ensure it happens without downtime and with new processes being able to register just fine:
+1. Run the first migration that adds the new column, nullable
+2. Deploy the updated Solid Queue code that uses this column
+2. Run the second migration. This migration does two things:
+  - Backfill existing rows that would have the column as NULL
+  - Make the column not nullable and add a new index
+
+Besides, it adds another migration with no effects to the `solid_queue_recurring_tasks` table. This one can be run just fine whenever, as the column affected is not used.
+
+To install the migrations:
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then follow the steps above, running first one, then deploying the code, then running the second one.
+
+## New behaviour when workers are killed
+From this version onwards, when a worker is killed and the supervisor can detect that, it'll fail in-progress jobs claimed by that worker. For this to work correctly, you need to run the above migration and ensure you restart any supervisors you'd have. 
+
+
+# Upgrading to version 0.5.x
+This version includes a new migration to improve recurring tasks. To install it, just run:
+
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then run the migrations.
+
+
+# Upgrading to version 0.4.x
+This version introduced an _async_ mode (this mode has been removed in version 0.7.0) to run the supervisor and have all workers and dispatchers run as part of the same process as the supervisor, instead of separate, forked, processes. Together with this, we introduced some changes in how the supervisor is started. Prior this change, you could choose whether you wanted to run workers, dispatchers or both, by starting Solid Queue as `solid_queue:work` or `solid_queue:dispatch`. From version 0.4.0, the only option available is:
+
+```
+$ bundle exec rake solid_queue:start
+```
+Whether the supervisor starts workers, dispatchers or both will depend on your configuration. For example, if you don't configure any dispatchers, only workers will be started. That is, with this configuration:
+
+```yml
+production:
+  workers:
+    - queues: [ real_time, background ]
+      threads: 5
+      polling_interval: 0.1
+      processes: 3
+```
+the supervisor will run 3 workers, each one with 5 threads, and no supervisors. With this configuration:
+```yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+      concurrency_maintenance_interval: 300
+```
+the supervisor will run 1 dispatcher and no workers.
+
+
+# Upgrading to version 0.3.x
+This version introduced support for [recurring (cron-style) jobs](https://github.com/rails/solid_queue/blob/main/README.md#recurring-tasks), and it needs a new DB migration for it. To install it, just run:
+
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then run the migrations.

data/lib/generators/solid_queue/install/USAGE

@@ -7,3 +7,4 @@ Example:
     This will perform the following:
         Installs solid_queue migrations
         Replaces Active Job's adapter in environment configuration
+        Installs bin/jobs binstub to start the supervisor

data/lib/generators/solid_queue/install/install_generator.rb

@@ -3,19 +3,33 @@
 class SolidQueue::InstallGenerator < Rails::Generators::Base
   source_root File.expand_path("templates", __dir__)
 
-  class_option :skip_migrations, type: :boolean, default: nil, desc: "Skip migrations"
+  class_option :skip_adapter, type: :boolean, default: nil, desc: "Skip setting Solid Queue as the Active Job's adapter"
+  class_option :database, type: :string, default: nil, desc: "The database to use for migrations, if different from the primary one."
 
   def add_solid_queue
-    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
-      gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
+    unless options[:skip_adapter]
+      if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
+        say "Setting solid_queue as Active Job's queue adapter"
+        gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
+      end
     end
 
-    copy_file "config.yml", "config/solid_queue.yml"
+    if File.exist?("config/solid_queue.yml")
+      say "Skipping sample configuration as config/solid_queue.yml exists"
+    else
+      say "Copying sample configuration"
+      copy_file "config.yml", "config/solid_queue.yml"
+    end
+
+    say "Copying binstub"
+    copy_file "jobs", "bin/jobs"
+    chmod "bin/jobs", 0755 & ~File.umask, verbose: false
   end
 
   def create_migrations
-    unless options[:skip_migrations]
-      rails_command "railties:install:migrations FROM=solid_queue", inline: true
-    end
+    say "Installing database migrations"
+    arguments = [ "FROM=solid_queue" ]
+    arguments << "DATABASE=#{options[:database]}" if options[:database].present?
+    rails_command "railties:install:migrations #{arguments.join(" ")}", inline: true
   end
 end

data/lib/generators/solid_queue/install/templates/jobs

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require_relative "../config/environment"
+require "solid_queue/cli"
+
+SolidQueue::Cli.start(ARGV)

data/lib/puma/plugin/solid_queue.rb

@@ -1,13 +1,5 @@
 require "puma/plugin"
 
-module Puma
-  class DSL
-    def solid_queue_mode(mode = :fork)
-      @options[:solid_queue_mode] = mode.to_sym
-    end
-  end
-end
-
 Puma::Plugin.create do
   attr_reader :puma_pid, :solid_queue_pid, :log_writer, :solid_queue_supervisor
 
@@ -15,36 +7,22 @@ Puma::Plugin.create do
     @log_writer = launcher.log_writer
     @puma_pid = $$
 
-    if launcher.options[:solid_queue_mode] == :async
-      start_async(launcher)
-    else
-      start_forked(launcher)
+    in_background do
+      monitor_solid_queue
     end
-  end
 
-  private
-    def start_forked(launcher)
-      in_background do
-        monitor_solid_queue
+    launcher.events.on_booted do
+      @solid_queue_pid = fork do
+        Thread.new { monitor_puma }
+        SolidQueue::Supervisor.start
       end
-
-      launcher.events.on_booted do
-        @solid_queue_pid = fork do
-          Thread.new { monitor_puma }
-          SolidQueue::Supervisor.start(mode: :fork)
-        end
-      end
-
-      launcher.events.on_stopped { stop_solid_queue }
-      launcher.events.on_restart { stop_solid_queue }
     end
 
-    def start_async(launcher)
-      launcher.events.on_booted { @solid_queue_supervisor = SolidQueue::Supervisor.start(mode: :async) }
-      launcher.events.on_stopped { solid_queue_supervisor.stop }
-      launcher.events.on_restart { solid_queue_supervisor.stop; solid_queue_supervisor.start }
-    end
+    launcher.events.on_stopped { stop_solid_queue }
+    launcher.events.on_restart { stop_solid_queue }
+  end
 
+  private
     def stop_solid_queue
       Process.waitpid(solid_queue_pid, Process::WNOHANG)
       log "Stopping Solid Queue..."

data/lib/solid_queue/cli.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require "thor"
+
+module SolidQueue
+  class Cli < Thor
+    class_option :config_file, type: :string, aliases: "-c", default: Configuration::DEFAULT_CONFIG_FILE_PATH, desc: "Path to config file"
+
+    def self.exit_on_failure?
+      true
+    end
+
+    desc :start, "Starts Solid Queue supervisor to dispatch and perform enqueued jobs. Default command."
+    default_command :start
+
+    def start
+      SolidQueue::Supervisor.start(load_configuration_from: options["config_file"])
+    end
+  end
+end

data/lib/solid_queue/configuration.rb

@@ -28,8 +28,7 @@ module SolidQueue
       dispatchers: [ DISPATCHER_DEFAULTS ]
     }
 
-    def initialize(mode: :fork, load_from: nil)
-      @mode = mode.to_s.inquiry
+    def initialize(load_from: nil)
       @raw_config = config_from(load_from)
     end
 
@@ -43,17 +42,13 @@ module SolidQueue
     end
 
     private
-      attr_reader :raw_config, :mode
+      attr_reader :raw_config
 
       DEFAULT_CONFIG_FILE_PATH = "config/solid_queue.yml"
 
       def workers
         workers_options.flat_map do |worker_options|
-          processes = if mode.fork?
-            worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
-          else
-            WORKER_DEFAULTS[:processes]
-          end
+          processes = worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
           processes.times.map { Process.new(:worker, worker_options.with_defaults(WORKER_DEFAULTS)) }
         end
       end

data/lib/solid_queue/dispatcher.rb

@@ -55,7 +55,7 @@ module SolidQueue
       end
 
       def set_procline
-        procline "waiting"
+        procline "dispatching every #{polling_interval.seconds} seconds"
       end
   end
 end

data/lib/solid_queue/lifecycle_hooks.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module LifecycleHooks
+    extend ActiveSupport::Concern
+
+    included do
+      mattr_reader :lifecycle_hooks, default: { start: [], stop: [] }
+    end
+
+    class_methods do
+      def on_start(&block)
+        self.lifecycle_hooks[:start] << block
+      end
+
+      def on_stop(&block)
+        self.lifecycle_hooks[:stop] << block
+      end
+
+      def clear_hooks
+        self.lifecycle_hooks[:start] = []
+        self.lifecycle_hooks[:stop] = []
+      end
+    end
+
+    private
+      def run_start_hooks
+        run_hooks_for :start
+      end
+
+      def run_stop_hooks
+        run_hooks_for :stop
+      end
+
+      def run_hooks_for(event)
+        self.class.lifecycle_hooks.fetch(event, []).each do |block|
+          block.call
+        rescue Exception => exception
+          handle_thread_error(exception)
+        end
+      end
+  end
+end

data/lib/solid_queue/processes/process_exit_error.rb

@@ -4,10 +4,13 @@ module SolidQueue
   module Processes
     class ProcessExitError < RuntimeError
       def initialize(status)
-        message = case
-        when status.exitstatus.present? then "Process pid=#{status.pid} exited with status #{status.  exitstatus}"
-        when status.signaled? then "Process pid=#{status.pid} received unhandled signal #{status. termsig}"
-        else "Process pid=#{status.pid} exited unexpectedly"
+        message = "Process pid=#{status.pid} exited unexpectedly."
+        if status.exitstatus.present?
+          message += " Exited with status #{status.exitstatus}."
+        end
+
+        if status.signaled?
+          message += " Received unhandled signal #{status.termsig}."
         end
 
         super(message)

data/lib/solid_queue/processes/runnable.rb

@@ -7,11 +7,7 @@ module SolidQueue::Processes
     attr_writer :mode
 
     def start
-      @stopped = false
-
-      SolidQueue.instrument(:start_process, process: self) do
-        run_callbacks(:boot) { boot }
-      end
+      boot
 
       if running_async?
         @thread = create_thread { run }
@@ -25,10 +21,6 @@ module SolidQueue::Processes
       @thread&.join
     end
 
-    def alive?
-      !running_async? || @thread.alive?
-    end
-
     private
       DEFAULT_MODE = :async
 
@@ -37,9 +29,15 @@ module SolidQueue::Processes
       end
 
       def boot
-        if running_as_fork?
-          register_signal_handlers
-          set_procline
+        SolidQueue.instrument(:start_process, process: self) do
+          run_callbacks(:boot) do
+            @stopped = false
+
+            if running_as_fork?
+              register_signal_handlers
+              set_procline
+            end
+          end
         end
       end
 

data/lib/solid_queue/supervisor.rb

@@ -2,16 +2,16 @@
 
 module SolidQueue
   class Supervisor < Processes::Base
-    include Maintenance
+    include LifecycleHooks
+    include Maintenance, Signals, Pidfiled
 
     class << self
-      def start(mode: :fork, load_configuration_from: nil)
+      def start(load_configuration_from: nil)
         SolidQueue.supervisor = true
-        configuration = Configuration.new(mode: mode, load_from: load_configuration_from)
+        configuration = Configuration.new(load_from: load_configuration_from)
 
         if configuration.configured_processes.any?
-          klass = mode == :fork ? ForkSupervisor : AsyncSupervisor
-          klass.new(configuration).tap(&:start)
+          new(configuration).tap(&:start)
         else
           abort "No workers or processed configured. Exiting..."
         end
@@ -20,11 +20,15 @@ module SolidQueue
 
     def initialize(configuration)
       @configuration = configuration
+      @forks = {}
+      @configured_processes = {}
+
       super
     end
 
     def start
       boot
+      run_start_hooks
 
       start_processes
       launch_maintenance_task
@@ -34,10 +38,11 @@ module SolidQueue
 
     def stop
       @stopped = true
+      run_stop_hooks
     end
 
     private
-      attr_reader :configuration
+      attr_reader :configuration, :forks, :configured_processes
 
       def boot
         SolidQueue.instrument(:start_process, process: self) do
@@ -52,15 +57,63 @@ module SolidQueue
         configuration.configured_processes.each { |configured_process| start_process(configured_process) }
       end
 
+      def supervise
+        loop do
+          break if stopped?
+
+          set_procline
+          process_signal_queue
+
+          unless stopped?
+            reap_and_replace_terminated_forks
+            interruptible_sleep(1.second)
+          end
+        end
+      ensure
+        shutdown
+      end
+
+      def start_process(configured_process)
+        process_instance = configured_process.instantiate.tap do |instance|
+          instance.supervised_by process
+          instance.mode = :fork
+        end
+
+        pid = fork do
+          process_instance.start
+        end
+
+        configured_processes[pid] = configured_process
+        forks[pid] = process_instance
+      end
+
       def stopped?
         @stopped
       end
 
-      def supervise
+      def set_procline
+        procline "supervising #{supervised_processes.join(", ")}"
       end
 
-      def start_process(configured_process)
-        raise NotImplementedError
+      def terminate_gracefully
+        SolidQueue.instrument(:graceful_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: supervised_processes) do |payload|
+          term_forks
+
+          Timer.wait_until(SolidQueue.shutdown_timeout, -> { all_forks_terminated? }) do
+            reap_terminated_forks
+          end
+
+          unless all_forks_terminated?
+            payload[:shutdown_timeout_exceeded] = true
+            terminate_immediately
+          end
+        end
+      end
+
+      def terminate_immediately
+        SolidQueue.instrument(:immediate_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: supervised_processes) do
+          quit_forks
+        end
       end
 
       def shutdown
@@ -74,5 +127,63 @@ module SolidQueue
       def sync_std_streams
         STDOUT.sync = STDERR.sync = true
       end
+
+      def supervised_processes
+        forks.keys
+      end
+
+      def term_forks
+        signal_processes(forks.keys, :TERM)
+      end
+
+      def quit_forks
+        signal_processes(forks.keys, :QUIT)
+      end
+
+      def reap_and_replace_terminated_forks
+        loop do
+          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
+          break unless pid
+
+          replace_fork(pid, status)
+        end
+      end
+
+      def reap_terminated_forks
+        loop do
+          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
+          break unless pid
+
+          if (terminated_fork = forks.delete(pid)) && (!status.exited? || status.exitstatus > 0)
+            handle_claimed_jobs_by(terminated_fork, status)
+          end
+
+          configured_processes.delete(pid)
+        end
+      rescue SystemCallError
+        # All children already reaped
+      end
+
+      def replace_fork(pid, status)
+        SolidQueue.instrument(:replace_fork, supervisor_pid: ::Process.pid, pid: pid, status: status) do |payload|
+          if terminated_fork = forks.delete(pid)
+            payload[:fork] = terminated_fork
+            handle_claimed_jobs_by(terminated_fork, status)
+
+            start_process(configured_processes.delete(pid))
+          end
+        end
+      end
+
+      def handle_claimed_jobs_by(terminated_fork, status)
+        if registered_process = process.supervisees.find_by(name: terminated_fork.name)
+          error = Processes::ProcessExitError.new(status)
+          registered_process.fail_all_claimed_executions_with(error)
+        end
+      end
+
+      def all_forks_terminated?
+        forks.empty?
+      end
   end
 end

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.6.1"
+  VERSION = "0.7.1"
 end

data/lib/solid_queue/worker.rb

@@ -2,6 +2,11 @@
 
 module SolidQueue
   class Worker < Processes::Poller
+    include LifecycleHooks
+
+    after_boot :run_start_hooks
+    before_shutdown :run_stop_hooks
+
     attr_accessor :queues, :pool
 
     def initialize(**options)

data/lib/solid_queue.rb

@@ -43,6 +43,16 @@ module SolidQueue
   mattr_accessor :clear_finished_jobs_after, default: 1.day
   mattr_accessor :default_concurrency_control_period, default: 3.minutes
 
+  delegate :on_start, :on_stop, to: Supervisor
+
+  def on_worker_start(...)
+    Worker.on_start(...)
+  end
+
+  def on_worker_stop(...)
+    Worker.on_stop(...)
+  end
+
   def supervisor?
     supervisor
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.6.1
+  version: 0.7.1
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-08-30 00:00:00.000000000 Z
+date: 2024-09-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -80,6 +80,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 1.11.0
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.3.1
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -188,6 +202,7 @@ files:
 - MIT-LICENSE
 - README.md
 - Rakefile
+- UPGRADING.md
 - app/jobs/solid_queue/recurring_job.rb
 - app/models/solid_queue/blocked_execution.rb
 - app/models/solid_queue/claimed_execution.rb
@@ -228,14 +243,17 @@ files:
 - lib/generators/solid_queue/install/USAGE
 - lib/generators/solid_queue/install/install_generator.rb
 - lib/generators/solid_queue/install/templates/config.yml
+- lib/generators/solid_queue/install/templates/jobs
 - lib/puma/plugin/solid_queue.rb
 - lib/solid_queue.rb
 - lib/solid_queue/app_executor.rb
+- lib/solid_queue/cli.rb
 - lib/solid_queue/configuration.rb
 - lib/solid_queue/dispatcher.rb
 - lib/solid_queue/dispatcher/concurrency_maintenance.rb
 - lib/solid_queue/dispatcher/recurring_schedule.rb
 - lib/solid_queue/engine.rb
+- lib/solid_queue/lifecycle_hooks.rb
 - lib/solid_queue/log_subscriber.rb
 - lib/solid_queue/pool.rb
 - lib/solid_queue/processes/base.rb
@@ -250,8 +268,6 @@ files:
 - lib/solid_queue/processes/runnable.rb
 - lib/solid_queue/processes/supervised.rb
 - lib/solid_queue/supervisor.rb
-- lib/solid_queue/supervisor/async_supervisor.rb
-- lib/solid_queue/supervisor/fork_supervisor.rb
 - lib/solid_queue/supervisor/maintenance.rb
 - lib/solid_queue/supervisor/pidfile.rb
 - lib/solid_queue/supervisor/pidfiled.rb
@@ -267,8 +283,9 @@ metadata:
   homepage_uri: https://github.com/rails/solid_queue
   source_code_uri: https://github.com/rails/solid_queue
 post_install_message: |
-  Upgrading to Solid Queue 0.4.x? There are some breaking changes about how Solid Queue is started. Check
-  https://github.com/rails/solid_queue/blob/main/UPGRADING.md for upgrade instructions.
+  Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
+  configuration and new migrations. Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
+  for upgrade instructions.
 rdoc_options: []
 require_paths:
 - lib

data/lib/solid_queue/supervisor/async_supervisor.rb

@@ -1,47 +0,0 @@
-# frozen_string_literal: true
-
-module SolidQueue
-  class Supervisor::AsyncSupervisor < Supervisor
-    def initialize(*)
-      super
-      @threads = Concurrent::Map.new
-    end
-
-    def kind
-      "Supervisor(async)"
-    end
-
-    def stop
-      super
-      stop_threads
-      threads.clear
-
-      shutdown
-    end
-
-    private
-      attr_reader :threads
-
-      def start_process(configured_process)
-        process_instance = configured_process.instantiate.tap do |instance|
-          instance.supervised_by process
-        end
-
-        process_instance.start
-
-        threads[process_instance.name] = process_instance
-      end
-
-      def stop_threads
-        stop_threads = threads.values.map do |thr|
-          Thread.new { thr.stop }
-        end
-
-        stop_threads.each { |thr| thr.join(SolidQueue.shutdown_timeout) }
-      end
-
-      def all_threads_terminated?
-        threads.values.none?(&:alive?)
-      end
-  end
-end

data/lib/solid_queue/supervisor/fork_supervisor.rb

@@ -1,126 +0,0 @@
-# frozen_string_literal: true
-
-module SolidQueue
-  class Supervisor::ForkSupervisor < Supervisor
-    include Signals, Pidfiled
-
-    def initialize(*)
-      super
-
-      @forks = {}
-      @configured_processes = {}
-    end
-
-    def kind
-      "Supervisor(fork)"
-    end
-
-    private
-      attr_reader :forks, :configured_processes
-
-      def supervise
-        loop do
-          break if stopped?
-
-          procline "supervising #{forks.keys.join(", ")}"
-          process_signal_queue
-
-          unless stopped?
-            reap_and_replace_terminated_forks
-            interruptible_sleep(1.second)
-          end
-        end
-      ensure
-        shutdown
-      end
-
-      def start_process(configured_process)
-        process_instance = configured_process.instantiate.tap do |instance|
-          instance.supervised_by process
-          instance.mode = :fork
-        end
-
-        pid = fork do
-          process_instance.start
-        end
-
-        configured_processes[pid] = configured_process
-        forks[pid] = process_instance
-      end
-
-      def terminate_gracefully
-        SolidQueue.instrument(:graceful_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: forks.keys) do |payload|
-          term_forks
-
-          Timer.wait_until(SolidQueue.shutdown_timeout, -> { all_forks_terminated? }) do
-            reap_terminated_forks
-          end
-
-          unless all_forks_terminated?
-            payload[:shutdown_timeout_exceeded] = true
-            terminate_immediately
-          end
-        end
-      end
-
-      def terminate_immediately
-        SolidQueue.instrument(:immediate_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: forks.keys) do
-          quit_forks
-        end
-      end
-
-      def term_forks
-        signal_processes(forks.keys, :TERM)
-      end
-
-      def quit_forks
-        signal_processes(forks.keys, :QUIT)
-      end
-
-      def reap_and_replace_terminated_forks
-        loop do
-          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
-          break unless pid
-
-          replace_fork(pid, status)
-        end
-      end
-
-      def reap_terminated_forks
-        loop do
-          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
-          break unless pid
-
-          if (terminated_fork = forks.delete(pid)) && !status.exited? || status.exitstatus > 0
-            handle_claimed_jobs_by(terminated_fork, status)
-          end
-
-          configured_processes.delete(pid)
-        end
-      rescue SystemCallError
-        # All children already reaped
-      end
-
-      def replace_fork(pid, status)
-        SolidQueue.instrument(:replace_fork, supervisor_pid: ::Process.pid, pid: pid, status: status) do |payload|
-          if terminated_fork = forks.delete(pid)
-            payload[:fork] = terminated_fork
-            handle_claimed_jobs_by(terminated_fork, status)
-
-            start_process(configured_processes.delete(pid))
-          end
-        end
-      end
-
-      def handle_claimed_jobs_by(terminated_fork, status)
-        if registered_process = process.supervisees.find_by(name: terminated_fork.name)
-          error = Processes::ProcessExitError.new(status)
-          registered_process.fail_all_claimed_executions_with(error)
-        end
-      end
-
-      def all_forks_terminated?
-        forks.empty?
-      end
-  end
-end