solid_queue 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e62ec09bbed1f2bcb96da96865c3516adbc254ad7cea234d3b6573ea001c758e
-  data.tar.gz: d78e7543194f3f6470bdb80c52ffca78c520a602d93423185c02ba82475bbda6
+  metadata.gz: 23836c755cda6dd9bec594148ad92ea19740da4125e9115d17c44bd2543d752c
+  data.tar.gz: 40d91182c13f155a86879f04abcf4232a50f3502f4038b5e9e07a3c15da8def2
 SHA512:
-  metadata.gz: e7cdceced9162911efdd0f2020042e45f7c73c74c506cf1d4958211b2cc34f02e96553d9910b3d3cca925f00e10960837be4872a33c3e44388e0b6c37cbcec31
-  data.tar.gz: d9a17282687ef9feaa4b357124411e9bd088bebb8c0ce6fd67ccf887e4d9cf20a431882e84b40f68ff5ca05a008258bf4d64b6ab86b675cb581be65154d1e39e
+  metadata.gz: f8ac683f1a0e635762d0d93b9b74f712c5fc5465381ea51230448dc2f176de67aafb0ee7031ae9e9cfee73edcee7bd70352e78aefd49ae25ea71a7485af95aff
+  data.tar.gz: b02eac304a30fb8c7e9e2627446a31bb56d22740e863b1d2749838429416f886da78d19d2000b533e3672576387ee075b8efb3c330f6177b5cd76a27955cd433

data/README.md

@@ -2,7 +2,7 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, and some way of specifying unique jobs are coming very soon._
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
@@ -31,9 +31,9 @@ $ bin/rails generate solid_queue:install
 
 This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
 
-Alternatively, you can add only the migration to your app:
+Alternatively, you can skip setting the Active Job's adapter with:
 ```bash
-$ bin/rails solid_queue:install:migrations
+$ bin/rails generate solid_queue:install --skip_adapter
 ```
 
 And set Solid Queue as your Active Job's queue backend manually, in your environment config:
@@ -42,7 +42,7 @@ And set Solid Queue as your Active Job's queue backend manually, in your environ
 config.active_job.queue_adapter = :solid_queue
 ```
 
-Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+Or you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
 
 ```ruby
 # app/jobs/my_job.rb
@@ -59,14 +59,14 @@ Finally, you need to run the migrations:
 $ bin/rails db:migrate
 ```
 
-After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them. You can use the provided binstub:`
 ```bash
-$ bundle exec rake solid_queue:start
+$ bin/jobs
 ```
 
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
-For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bundle exec rake solid_queue:start` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
+For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
 ## Requirements
 Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
@@ -80,7 +80,7 @@ We have three types of actors in Solid Queue:
 - _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They're also in charge of managing [recurring tasks](#recurring-tasks), dispatching jobs to process them according to their schedule. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
 - The _supervisor_ runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.
 
-By default, Solid Queue runs in `fork` mode. This means the supervisor will fork a separate process for each supervised worker/dispatcher. There's also an `async` mode where each worker and dispatcher will be run as a thread of the supervisor process. This can be used with [the provided Puma plugin](#puma-plugin)
+Solid Queue's supervisor will fork a separate process for each supervised worker/dispatcher.
 
 By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
 
@@ -131,7 +131,7 @@ Here's an overview of the different options:
 
   Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
 - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
-- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. **Note**: this option will be ignored if [running in `async` mode](#running-as-a-fork-or-asynchronously).
+- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 - `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
 
@@ -194,13 +194,13 @@ development:
     # ...
 ```
 
-Install migrations and specify the dedicated database name with the `DATABASE` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
+Install migrations and specify the dedicated database name with the `--database` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
 
 ```bash
-$ bin/rails solid_queue:install:migrations DATABASE=solid_queue
+$ bin/rails g solid_queue:install --database solid_queue
 ```
 
-Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`), the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
+Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`) without a `--database` option, the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
 
 Finally, run the migrations:
 
@@ -305,18 +305,6 @@ plugin :solid_queue
 ```
 to your `puma.rb` configuration.
 
-### Running as a fork or asynchronously
-
-By default, the Puma plugin will fork additional processes for each worker and dispatcher so that they run in different processes. This provides the best isolation and performance, but can have additional memory usage.
-
-Alternatively, workers and dispatchers can be run within the same Puma process(s). To do so just configure the plugin as:
-
-```ruby
-plugin :solid_queue
-solid_queue_mode :async
-```
-
-Note that in this case, the `processes` configuration option will be ignored.
 
 ## Jobs and transactional integrity
 :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.

data/UPGRADING.md

@@ -0,0 +1,102 @@
+# Upgrading to version 0.7.x
+
+This version removed the new async mode introduced in version 0.4.0 and introduced a new binstub that can be used to start Solid Queue's supervisor. It includes also a minor migration.
+
+To install both the binstub `bin/jobs` and the migration, you can just run
+```
+bin/rails generate solid_queue:install
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails generate solid_queue:install --database <the_name_of_your_solid_queue_db>
+```
+
+
+# Upgrading to version 0.6.x
+
+## New migration in 3 steps
+This version adds two new migrations to modify the `solid_queue_processes` table. The goal of that migration is to add a new column that needs to be `NOT NULL`. This needs to be done with two migrations and the following steps to ensure it happens without downtime and with new processes being able to register just fine:
+1. Run the first migration that adds the new column, nullable
+2. Deploy the updated Solid Queue code that uses this column
+2. Run the second migration. This migration does two things:
+  - Backfill existing rows that would have the column as NULL
+  - Make the column not nullable and add a new index
+
+Besides, it adds another migration with no effects to the `solid_queue_recurring_tasks` table. This one can be run just fine whenever, as the column affected is not used.
+
+To install the migrations:
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then follow the steps above, running first one, then deploying the code, then running the second one.
+
+## New behaviour when workers are killed
+From this version onwards, when a worker is killed and the supervisor can detect that, it'll fail in-progress jobs claimed by that worker. For this to work correctly, you need to run the above migration and ensure you restart any supervisors you'd have. 
+
+
+# Upgrading to version 0.5.x
+This version includes a new migration to improve recurring tasks. To install it, just run:
+
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then run the migrations.
+
+
+# Upgrading to version 0.4.x
+This version introduced an _async_ mode (this mode has been removed in version 0.7.0) to run the supervisor and have all workers and dispatchers run as part of the same process as the supervisor, instead of separate, forked, processes. Together with this, we introduced some changes in how the supervisor is started. Prior this change, you could choose whether you wanted to run workers, dispatchers or both, by starting Solid Queue as `solid_queue:work` or `solid_queue:dispatch`. From version 0.4.0, the only option available is:
+
+```
+$ bundle exec rake solid_queue:start
+```
+Whether the supervisor starts workers, dispatchers or both will depend on your configuration. For example, if you don't configure any dispatchers, only workers will be started. That is, with this configuration:
+
+```yml
+production:
+  workers:
+    - queues: [ real_time, background ]
+      threads: 5
+      polling_interval: 0.1
+      processes: 3
+```
+the supervisor will run 3 workers, each one with 5 threads, and no supervisors. With this configuration:
+```yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+      concurrency_maintenance_interval: 300
+```
+the supervisor will run 1 dispatcher and no workers.
+
+
+# Upgrading to version 0.3.x
+This version introduced support for [recurring (cron-style) jobs](https://github.com/rails/solid_queue/blob/main/README.md#recurring-tasks), and it needs a new DB migration for it. To install it, just run:
+
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then run the migrations.

data/app/models/solid_queue/claimed_execution.rb

@@ -29,8 +29,9 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     def release_all
       SolidQueue.instrument(:release_many_claimed) do |payload|
         includes(:job).tap do |executions|
-          payload[:size] = executions.size
           executions.each(&:release)
+
+          payload[:size] = executions.size
         end
       end
     end
@@ -38,11 +39,11 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     def fail_all_with(error)
       SolidQueue.instrument(:fail_many_claimed) do |payload|
         includes(:job).tap do |executions|
-          payload[:size] = executions.size
+          executions.each { |execution| execution.failed_with(error) }
+
           payload[:process_ids] = executions.map(&:process_id).uniq
           payload[:job_ids] = executions.map(&:job_id).uniq
-
-          executions.each { |execution| execution.failed_with(error) }
+          payload[:size] = executions.size
         end
       end
     end

data/app/models/solid_queue/process/executor.rb

@@ -8,7 +8,7 @@ module SolidQueue
       included do
         has_many :claimed_executions
 
-        after_destroy -> { claimed_executions.release_all }, if: :claims_executions?
+        after_destroy :release_all_claimed_executions
       end
 
       def fail_all_claimed_executions_with(error)
@@ -17,6 +17,12 @@ module SolidQueue
         end
       end
 
+      def release_all_claimed_executions
+        if claims_executions?
+          claimed_executions.release_all
+        end
+      end
+
       private
         def claims_executions?
           kind == "Worker"

data/app/models/solid_queue/process/prunable.rb

@@ -1,12 +1,6 @@
 # frozen_string_literal: true
 
 module SolidQueue
-  class ProcessPrunedError < RuntimeError
-    def initialize(last_heartbeat_at)
-      super("Process was found dead and pruned (last heartbeat at: #{last_heartbeat_at}")
-    end
-  end
-
   class Process
     module Prunable
       extend ActiveSupport::Concern
@@ -28,7 +22,7 @@ module SolidQueue
       end
 
       def prune
-        error = ProcessPrunedError.new(last_heartbeat_at)
+        error = Processes::ProcessPrunedError.new(last_heartbeat_at)
         fail_all_claimed_executions_with(error)
 
         deregister(pruned: true)

data/app/models/solid_queue/process.rb

@@ -4,7 +4,7 @@ class SolidQueue::Process < SolidQueue::Record
   include Executor, Prunable
 
   belongs_to :supervisor, class_name: "SolidQueue::Process", optional: true, inverse_of: :supervisees
-  has_many :supervisees, class_name: "SolidQueue::Process", inverse_of: :supervisor, foreign_key: :supervisor_id, dependent: :destroy
+  has_many :supervisees, class_name: "SolidQueue::Process", inverse_of: :supervisor, foreign_key: :supervisor_id
 
   store :metadata, coder: JSON
 
@@ -26,9 +26,18 @@ class SolidQueue::Process < SolidQueue::Record
   def deregister(pruned: false)
     SolidQueue.instrument :deregister_process, process: self, pruned: pruned do |payload|
       destroy!
+
+      unless supervised? || pruned
+        supervisees.each(&:deregister)
+      end
     rescue Exception => error
       payload[:error] = error
       raise
     end
   end
+
+  private
+    def supervised?
+      supervisor_id.present?
+    end
 end

data/db/migrate/20240813160053_make_name_not_null.rb

@@ -11,6 +11,6 @@ class MakeNameNotNull < ActiveRecord::Migration[7.1]
 
   def down
     remove_index :solid_queue_processes, [ :name, :supervisor_id ]
-    change_column :solid_queue_processes, :name, :string, null: false
+    change_column :solid_queue_processes, :name, :string, null: true
   end
 end

data/lib/generators/solid_queue/install/USAGE

@@ -7,3 +7,4 @@ Example:
     This will perform the following:
         Installs solid_queue migrations
         Replaces Active Job's adapter in environment configuration
+        Installs bin/jobs binstub to start the supervisor

data/lib/generators/solid_queue/install/install_generator.rb

@@ -3,19 +3,33 @@
 class SolidQueue::InstallGenerator < Rails::Generators::Base
   source_root File.expand_path("templates", __dir__)
 
-  class_option :skip_migrations, type: :boolean, default: nil, desc: "Skip migrations"
+  class_option :skip_adapter, type: :boolean, default: nil, desc: "Skip setting Solid Queue as the Active Job's adapter"
+  class_option :database, type: :string, default: nil, desc: "The database to use for migrations, if different from the primary one."
 
   def add_solid_queue
-    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
-      gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
+    unless options[:skip_adapter]
+      if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
+        say "Setting solid_queue as Active Job's queue adapter"
+        gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
+      end
     end
 
-    copy_file "config.yml", "config/solid_queue.yml"
+    if File.exist?("config/solid_queue.yml")
+      say "Skipping sample configuration as config/solid_queue.yml exists"
+    else
+      say "Copying sample configuration"
+      copy_file "config.yml", "config/solid_queue.yml"
+    end
+
+    say "Copying binstub"
+    copy_file "jobs", "bin/jobs"
+    chmod "bin/jobs", 0755 & ~File.umask, verbose: false
   end
 
   def create_migrations
-    unless options[:skip_migrations]
-      rails_command "railties:install:migrations FROM=solid_queue", inline: true
-    end
+    say "Installing database migrations"
+    arguments = [ "FROM=solid_queue" ]
+    arguments << "DATABASE=#{options[:database]}" if options[:database].present?
+    rails_command "railties:install:migrations #{arguments.join(" ")}", inline: true
   end
 end

data/lib/generators/solid_queue/install/templates/jobs

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require_relative "../config/environment"
+require "solid_queue/cli"
+
+SolidQueue::Cli.start(ARGV)

data/lib/puma/plugin/solid_queue.rb

@@ -1,13 +1,5 @@
 require "puma/plugin"
 
-module Puma
-  class DSL
-    def solid_queue_mode(mode = :fork)
-      @options[:solid_queue_mode] = mode.to_sym
-    end
-  end
-end
-
 Puma::Plugin.create do
   attr_reader :puma_pid, :solid_queue_pid, :log_writer, :solid_queue_supervisor
 
@@ -15,36 +7,22 @@ Puma::Plugin.create do
     @log_writer = launcher.log_writer
     @puma_pid = $$
 
-    if launcher.options[:solid_queue_mode] == :async
-      start_async(launcher)
-    else
-      start_forked(launcher)
+    in_background do
+      monitor_solid_queue
     end
-  end
 
-  private
-    def start_forked(launcher)
-      in_background do
-        monitor_solid_queue
+    launcher.events.on_booted do
+      @solid_queue_pid = fork do
+        Thread.new { monitor_puma }
+        SolidQueue::Supervisor.start
       end
-
-      launcher.events.on_booted do
-        @solid_queue_pid = fork do
-          Thread.new { monitor_puma }
-          SolidQueue::Supervisor.start(mode: :fork)
-        end
-      end
-
-      launcher.events.on_stopped { stop_solid_queue }
-      launcher.events.on_restart { stop_solid_queue }
     end
 
-    def start_async(launcher)
-      launcher.events.on_booted { @solid_queue_supervisor = SolidQueue::Supervisor.start(mode: :async) }
-      launcher.events.on_stopped { solid_queue_supervisor.stop }
-      launcher.events.on_restart { solid_queue_supervisor.stop; solid_queue_supervisor.start }
-    end
+    launcher.events.on_stopped { stop_solid_queue }
+    launcher.events.on_restart { stop_solid_queue }
+  end
 
+  private
     def stop_solid_queue
       Process.waitpid(solid_queue_pid, Process::WNOHANG)
       log "Stopping Solid Queue..."

data/lib/solid_queue/cli.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require "thor"
+
+module SolidQueue
+  class Cli < Thor
+    class_option :config_file, type: :string, aliases: "-c", default: Configuration::DEFAULT_CONFIG_FILE_PATH, desc: "Path to config file"
+
+    def self.exit_on_failure?
+      true
+    end
+
+    desc :start, "Starts Solid Queue supervisor to dispatch and perform enqueued jobs. Default command."
+    default_command :start
+
+    def start
+      SolidQueue::Supervisor.start(load_configuration_from: options["config_file"])
+    end
+  end
+end

data/lib/solid_queue/configuration.rb

@@ -23,8 +23,12 @@ module SolidQueue
       recurring_tasks: []
     }
 
-    def initialize(mode: :fork, load_from: nil)
-      @mode = mode.to_s.inquiry
+    DEFAULT_CONFIG = {
+      workers: [ WORKER_DEFAULTS ],
+      dispatchers: [ DISPATCHER_DEFAULTS ]
+    }
+
+    def initialize(load_from: nil)
       @raw_config = config_from(load_from)
     end
 
@@ -38,17 +42,13 @@ module SolidQueue
     end
 
     private
-      attr_reader :raw_config, :mode
+      attr_reader :raw_config
 
       DEFAULT_CONFIG_FILE_PATH = "config/solid_queue.yml"
 
       def workers
         workers_options.flat_map do |worker_options|
-          processes = if mode.fork?
-            worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
-          else
-            WORKER_DEFAULTS[:processes]
-          end
+          processes = worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
           processes.times.map { Process.new(:worker, worker_options.with_defaults(WORKER_DEFAULTS)) }
         end
       end
@@ -61,22 +61,27 @@ module SolidQueue
       end
 
       def config_from(file_or_hash, env: Rails.env)
-        config = load_config_from(file_or_hash)
-        config[env.to_sym] ? config[env.to_sym] : config
+        load_config_from(file_or_hash).then do |config|
+          config = config[env.to_sym] ? config[env.to_sym] : config
+          if (config.keys & DEFAULT_CONFIG.keys).any? then config
+          else
+            DEFAULT_CONFIG
+          end
+        end
       end
 
       def workers_options
-        @workers_options ||= options_from_raw_config(:workers, WORKER_DEFAULTS)
+        @workers_options ||= options_from_raw_config(:workers)
           .map { |options| options.dup.symbolize_keys }
       end
 
       def dispatchers_options
-        @dispatchers_options ||= options_from_raw_config(:dispatchers, DISPATCHER_DEFAULTS)
+        @dispatchers_options ||= options_from_raw_config(:dispatchers)
           .map { |options| options.dup.symbolize_keys }
       end
 
-      def options_from_raw_config(key, defaults)
-        raw_config.empty? ? [ defaults ] : Array(raw_config[key])
+      def options_from_raw_config(key)
+        Array(raw_config[key])
       end
 
       def parse_recurring_tasks(tasks)

data/lib/solid_queue/dispatcher.rb

@@ -55,7 +55,7 @@ module SolidQueue
       end
 
       def set_procline
-        procline "waiting"
+        procline "dispatching every #{polling_interval.seconds} seconds"
       end
   end
 end

data/lib/solid_queue/processes/poller.rb

@@ -45,10 +45,12 @@ module SolidQueue::Processes
       end
 
       def with_polling_volume
-        if SolidQueue.silence_polling? && ActiveRecord::Base.logger
-          ActiveRecord::Base.logger.silence { yield }
-        else
-          yield
+        SolidQueue.instrument(:polling) do
+          if SolidQueue.silence_polling? && ActiveRecord::Base.logger
+            ActiveRecord::Base.logger.silence { yield }
+          else
+            yield
+          end
         end
       end
   end

data/lib/solid_queue/processes/process_exit_error.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module Processes
+    class ProcessExitError < RuntimeError
+      def initialize(status)
+        message = "Process pid=#{status.pid} exited unexpectedly."
+        if status.exitstatus.present?
+          message += " Exited with status #{status.exitstatus}."
+        end
+
+        if status.signaled?
+          message += " Received unhandled signal #{status.termsig}."
+        end
+
+        super(message)
+      end
+    end
+  end
+end

data/lib/solid_queue/processes/process_missing_error.rb

@@ -0,0 +1,9 @@
+module SolidQueue
+  module Processes
+    class ProcessMissingError < RuntimeError
+      def initialize
+        super("The process that was running this job no longer exists")
+      end
+    end
+  end
+end

data/lib/solid_queue/processes/process_pruned_error.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module Processes
+    class ProcessPrunedError < RuntimeError
+      def initialize(last_heartbeat_at)
+        super("Process was found dead and pruned (last heartbeat at: #{last_heartbeat_at}")
+      end
+    end
+  end
+end

data/lib/solid_queue/supervisor/maintenance.rb

@@ -1,10 +1,4 @@
 module SolidQueue
-  class ProcessMissingError < RuntimeError
-    def initialize
-      super("The process that was running this job no longer exists")
-    end
-  end
-
   module Supervisor::Maintenance
     extend ActiveSupport::Concern
 
@@ -35,7 +29,7 @@ module SolidQueue
 
       def fail_orphaned_executions
         wrap_in_app_executor do
-          SolidQueue::ClaimedExecution.orphaned.fail_all_with(ProcessMissingError.new)
+          ClaimedExecution.orphaned.fail_all_with(Processes::ProcessMissingError.new)
         end
       end
   end

data/lib/solid_queue/supervisor.rb

@@ -2,20 +2,26 @@
 
 module SolidQueue
   class Supervisor < Processes::Base
-    include Maintenance
+    include Maintenance, Signals, Pidfiled
 
     class << self
-      def start(mode: :fork, load_configuration_from: nil)
+      def start(load_configuration_from: nil)
         SolidQueue.supervisor = true
-        configuration = Configuration.new(mode: mode, load_from: load_configuration_from)
+        configuration = Configuration.new(load_from: load_configuration_from)
 
-        klass = mode == :fork ? ForkSupervisor : AsyncSupervisor
-        klass.new(configuration).tap(&:start)
+        if configuration.configured_processes.any?
+          new(configuration).tap(&:start)
+        else
+          abort "No workers or processed configured. Exiting..."
+        end
       end
     end
 
     def initialize(configuration)
       @configuration = configuration
+      @forks = {}
+      @configured_processes = {}
+
       super
     end
 
@@ -33,7 +39,7 @@ module SolidQueue
     end
 
     private
-      attr_reader :configuration
+      attr_reader :configuration, :forks, :configured_processes
 
       def boot
         SolidQueue.instrument(:start_process, process: self) do
@@ -48,15 +54,63 @@ module SolidQueue
         configuration.configured_processes.each { |configured_process| start_process(configured_process) }
       end
 
+      def supervise
+        loop do
+          break if stopped?
+
+          set_procline
+          process_signal_queue
+
+          unless stopped?
+            reap_and_replace_terminated_forks
+            interruptible_sleep(1.second)
+          end
+        end
+      ensure
+        shutdown
+      end
+
+      def start_process(configured_process)
+        process_instance = configured_process.instantiate.tap do |instance|
+          instance.supervised_by process
+          instance.mode = :fork
+        end
+
+        pid = fork do
+          process_instance.start
+        end
+
+        configured_processes[pid] = configured_process
+        forks[pid] = process_instance
+      end
+
       def stopped?
         @stopped
       end
 
-      def supervise
+      def set_procline
+        procline "supervising #{supervised_processes.join(", ")}"
       end
 
-      def start_process(configured_process)
-        raise NotImplementedError
+      def terminate_gracefully
+        SolidQueue.instrument(:graceful_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: supervised_processes) do |payload|
+          term_forks
+
+          Timer.wait_until(SolidQueue.shutdown_timeout, -> { all_forks_terminated? }) do
+            reap_terminated_forks
+          end
+
+          unless all_forks_terminated?
+            payload[:shutdown_timeout_exceeded] = true
+            terminate_immediately
+          end
+        end
+      end
+
+      def terminate_immediately
+        SolidQueue.instrument(:immediate_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: supervised_processes) do
+          quit_forks
+        end
       end
 
       def shutdown
@@ -70,5 +124,63 @@ module SolidQueue
       def sync_std_streams
         STDOUT.sync = STDERR.sync = true
       end
+
+      def supervised_processes
+        forks.keys
+      end
+
+      def term_forks
+        signal_processes(forks.keys, :TERM)
+      end
+
+      def quit_forks
+        signal_processes(forks.keys, :QUIT)
+      end
+
+      def reap_and_replace_terminated_forks
+        loop do
+          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
+          break unless pid
+
+          replace_fork(pid, status)
+        end
+      end
+
+      def reap_terminated_forks
+        loop do
+          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
+          break unless pid
+
+          if (terminated_fork = forks.delete(pid)) && (!status.exited? || status.exitstatus > 0)
+            handle_claimed_jobs_by(terminated_fork, status)
+          end
+
+          configured_processes.delete(pid)
+        end
+      rescue SystemCallError
+        # All children already reaped
+      end
+
+      def replace_fork(pid, status)
+        SolidQueue.instrument(:replace_fork, supervisor_pid: ::Process.pid, pid: pid, status: status) do |payload|
+          if terminated_fork = forks.delete(pid)
+            payload[:fork] = terminated_fork
+            handle_claimed_jobs_by(terminated_fork, status)
+
+            start_process(configured_processes.delete(pid))
+          end
+        end
+      end
+
+      def handle_claimed_jobs_by(terminated_fork, status)
+        if registered_process = process.supervisees.find_by(name: terminated_fork.name)
+          error = Processes::ProcessExitError.new(status)
+          registered_process.fail_all_claimed_executions_with(error)
+        end
+      end
+
+      def all_forks_terminated?
+        forks.empty?
+      end
   end
 end

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.6.0"
+  VERSION = "0.7.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-08-21 00:00:00.000000000 Z
+date: 2024-09-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -80,6 +80,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 1.11.0
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.3.1
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -188,6 +202,7 @@ files:
 - MIT-LICENSE
 - README.md
 - Rakefile
+- UPGRADING.md
 - app/jobs/solid_queue/recurring_job.rb
 - app/models/solid_queue/blocked_execution.rb
 - app/models/solid_queue/claimed_execution.rb
@@ -228,9 +243,11 @@ files:
 - lib/generators/solid_queue/install/USAGE
 - lib/generators/solid_queue/install/install_generator.rb
 - lib/generators/solid_queue/install/templates/config.yml
+- lib/generators/solid_queue/install/templates/jobs
 - lib/puma/plugin/solid_queue.rb
 - lib/solid_queue.rb
 - lib/solid_queue/app_executor.rb
+- lib/solid_queue/cli.rb
 - lib/solid_queue/configuration.rb
 - lib/solid_queue/dispatcher.rb
 - lib/solid_queue/dispatcher/concurrency_maintenance.rb
@@ -242,13 +259,14 @@ files:
 - lib/solid_queue/processes/callbacks.rb
 - lib/solid_queue/processes/interruptible.rb
 - lib/solid_queue/processes/poller.rb
+- lib/solid_queue/processes/process_exit_error.rb
+- lib/solid_queue/processes/process_missing_error.rb
+- lib/solid_queue/processes/process_pruned_error.rb
 - lib/solid_queue/processes/procline.rb
 - lib/solid_queue/processes/registrable.rb
 - lib/solid_queue/processes/runnable.rb
 - lib/solid_queue/processes/supervised.rb
 - lib/solid_queue/supervisor.rb
-- lib/solid_queue/supervisor/async_supervisor.rb
-- lib/solid_queue/supervisor/fork_supervisor.rb
 - lib/solid_queue/supervisor/maintenance.rb
 - lib/solid_queue/supervisor/pidfile.rb
 - lib/solid_queue/supervisor/pidfiled.rb
@@ -264,8 +282,9 @@ metadata:
   homepage_uri: https://github.com/rails/solid_queue
   source_code_uri: https://github.com/rails/solid_queue
 post_install_message: |
-  Upgrading to Solid Queue 0.4.x? There are some breaking changes about how Solid Queue is started. Check
-  https://github.com/rails/solid_queue/blob/main/UPGRADING.md for upgrade instructions.
+  Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
+  configuration and new migrations. Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
+  for upgrade instructions.
 rdoc_options: []
 require_paths:
 - lib

data/lib/solid_queue/supervisor/async_supervisor.rb

@@ -1,47 +0,0 @@
-# frozen_string_literal: true
-
-module SolidQueue
-  class Supervisor::AsyncSupervisor < Supervisor
-    def initialize(*)
-      super
-      @threads = Concurrent::Map.new
-    end
-
-    def kind
-      "Supervisor(async)"
-    end
-
-    def stop
-      super
-      stop_threads
-      threads.clear
-
-      shutdown
-    end
-
-    private
-      attr_reader :threads
-
-      def start_process(configured_process)
-        process_instance = configured_process.instantiate.tap do |instance|
-          instance.supervised_by process
-        end
-
-        process_instance.start
-
-        threads[process_instance.name] = process_instance
-      end
-
-      def stop_threads
-        stop_threads = threads.values.map do |thr|
-          Thread.new { thr.stop }
-        end
-
-        stop_threads.each { |thr| thr.join(SolidQueue.shutdown_timeout) }
-      end
-
-      def all_threads_terminated?
-        threads.values.none?(&:alive?)
-      end
-  end
-end

data/lib/solid_queue/supervisor/fork_supervisor.rb

@@ -1,138 +0,0 @@
-# frozen_string_literal: true
-
-module SolidQueue
-  class ProcessExitError < RuntimeError
-    def initialize(status)
-      message = case
-      when status.exitstatus.present? then "Process pid=#{status.pid} exited with status #{status.exitstatus}"
-      when status.signaled? then "Process pid=#{status.pid} received unhandled signal #{status.termsig}"
-      else "Process pid=#{status.pid} exited unexpectedly"
-      end
-
-      super(message)
-    end
-  end
-
-  class Supervisor::ForkSupervisor < Supervisor
-    include Signals, Pidfiled
-
-    def initialize(*)
-      super
-
-      @forks = {}
-      @configured_processes = {}
-    end
-
-    def kind
-      "Supervisor(fork)"
-    end
-
-    private
-      attr_reader :forks, :configured_processes
-
-      def supervise
-        loop do
-          break if stopped?
-
-          procline "supervising #{forks.keys.join(", ")}"
-          process_signal_queue
-
-          unless stopped?
-            reap_and_replace_terminated_forks
-            interruptible_sleep(1.second)
-          end
-        end
-      ensure
-        shutdown
-      end
-
-      def start_process(configured_process)
-        process_instance = configured_process.instantiate.tap do |instance|
-          instance.supervised_by process
-          instance.mode = :fork
-        end
-
-        pid = fork do
-          process_instance.start
-        end
-
-        configured_processes[pid] = configured_process
-        forks[pid] = process_instance
-      end
-
-      def terminate_gracefully
-        SolidQueue.instrument(:graceful_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: forks.keys) do |payload|
-          term_forks
-
-          Timer.wait_until(SolidQueue.shutdown_timeout, -> { all_forks_terminated? }) do
-            reap_terminated_forks
-          end
-
-          unless all_forks_terminated?
-            payload[:shutdown_timeout_exceeded] = true
-            terminate_immediately
-          end
-        end
-      end
-
-      def terminate_immediately
-        SolidQueue.instrument(:immediate_termination, process_id: process_id, supervisor_pid: ::Process.pid, supervised_processes: forks.keys) do
-          quit_forks
-        end
-      end
-
-      def term_forks
-        signal_processes(forks.keys, :TERM)
-      end
-
-      def quit_forks
-        signal_processes(forks.keys, :QUIT)
-      end
-
-      def reap_and_replace_terminated_forks
-        loop do
-          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
-          break unless pid
-
-          replace_fork(pid, status)
-        end
-      end
-
-      def reap_terminated_forks
-        loop do
-          pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
-          break unless pid
-
-          if (terminated_fork = forks.delete(pid)) && !status.exited? || status.exitstatus > 0
-            handle_claimed_jobs_by(terminated_fork, status)
-          end
-
-          configured_processes.delete(pid)
-        end
-      rescue SystemCallError
-        # All children already reaped
-      end
-
-      def replace_fork(pid, status)
-        SolidQueue.instrument(:replace_fork, supervisor_pid: ::Process.pid, pid: pid, status: status) do |payload|
-          if terminated_fork = forks.delete(pid)
-            payload[:fork] = terminated_fork
-            handle_claimed_jobs_by(terminated_fork, status)
-
-            start_process(configured_processes.delete(pid))
-          end
-        end
-      end
-
-      def handle_claimed_jobs_by(terminated_fork, status)
-        if registered_process = process.supervisees.find_by(name: terminated_fork.name)
-          error = ProcessExitError.new(status)
-          registered_process.fail_all_claimed_executions_with(error)
-        end
-      end
-
-      def all_forks_terminated?
-        forks.empty?
-      end
-  end
-end