solid_queue 1.5.0 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d40f055103084c9aaa9efe6ded0e675ff967a88200bf9f5579b1287e29d48818
-  data.tar.gz: af6ab85d5c674dbceff32768b5aa7b4d87bc1aeea24476e3fb469d639c4fad46
+  metadata.gz: d6cda6edb92a7805d6c2c0fc13b2649c2c2df26a4d65e555049efde022cb8d14
+  data.tar.gz: c6d0036cbeb743e56faa4ed9fe6756f1a8e3ae3705a402f55440bb14cb3174dc
 SHA512:
-  metadata.gz: 926a7f82ebca8389d5d6d6116bd6539d65715704e739f66083f1e21411737da72972f4b643bd4291e95ef5bb107417cb11fde6c535d6e4013434ac589cc2fc6e
-  data.tar.gz: 684e5f32bbe7aa4ce0645687a92c2538016431151208156ec144b574e1d48e8d04144f28a2eacca86ff5bd4449a2c6bdd439bdcd74bdf72788ae4430abd284c2
+  metadata.gz: 5f7420684f8aa122d12e2314aa2445fb31a3f18f2a7bc53b4ceffccccd131a7f6d9408ef776da33dbbfe437be4c835437524279ac6150e57d134cf85698caf7a
+  data.tar.gz: cd02de9753ce345ec599d72230f634df0e633f278871df5da4330318fcb92b8820d99612f43fb8d89be348460d1cb198a5bfab34fdce5c6f639709ab8dd426a2

data/README.md

@@ -204,6 +204,10 @@ Or you can also set the environment variable `SOLID_QUEUE_SUPERVISOR_MODE` to `a
 
 **The recommended and default mode is `fork`. Only use `async` if you know what you're doing and have strong reasons to**
 
+This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (`threads: N`) or as fibers on a single fiber reactor thread (`fibers: N`).
+
+Because these are separate concerns, you can combine the default `fork` supervisor mode with fiber workers. In that setup, each worker process gets its own fiber reactor and bounded fiber count.
+
 ## Configuration
 
 By default, Solid Queue will try to find your configuration under `config/queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG` or by using the `-c/--config_file` option with `bin/jobs`, like this:
@@ -214,6 +218,8 @@ bin/jobs -c config/calendar.yml
 
 You can also skip the scheduler process by setting the environment variable `SOLID_QUEUE_SKIP_RECURRING=true`. This is useful for environments like staging, review apps, or development where you don't want any recurring jobs to run. This is equivalent to using the `--skip-recurring` option with `bin/jobs`.
 
+To run **only** the scheduler (no workers or dispatchers)—for example to isolate recurring tasks on a dedicated process—set `SOLID_QUEUE_ONLY_RECURRING=true` or use the `--only-recurring` option with `bin/jobs`.
+
 This is what this configuration looks like:
 
 ```yml
@@ -230,6 +236,9 @@ production:
       threads: 5
       polling_interval: 0.1
       processes: 3
+    - queues: "api*"
+      fibers: 100
+      polling_interval: 0.05
   scheduler:
     dynamic_tasks_enabled: true
     polling_interval: 5
@@ -272,9 +281,11 @@ Here's an overview of the different options:
 
   Check the sections below on [how queue order behaves combined with priorities](#queue-order-and-priorities), and [how the way you specify the queues per worker might affect performance](#queues-specification-and-performance).
 
-- `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
-It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker thread uses one connection, and two additional connections are reserved for polling and heartbeat.
-- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. **Note**: this option will be ignored if [running in `async` mode](#fork-vs-async-mode).
+- `threads`: configures a worker to execute jobs in a thread pool of this size. By default, workers use `threads: 3`. Only workers have this setting, and it can't be combined with `fibers`.
+It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker uses connections for polling and heartbeat and thread mode may use additional connections for job execution.
+- `fibers`: configures a worker to execute jobs as fibers on a single fiber reactor thread, with this value as the maximum number of in-flight jobs. It can't be combined with `threads`.
+  Fiber workers require fiber-scoped isolated execution state. In Rails apps, set `config.active_support.isolation_level = :fiber` before using `fibers`. Solid Queue refuses to boot fiber workers when isolation remains thread-scoped. On Rails 7.2 and later, a practical starting point is usually `3-5` queue database connections per worker process rather than matching the `fibers` value, because ordinary Active Record query paths can release connections between non-blocking waits. On Rails 7.1, size the queue database pool more conservatively, as in-flight fiber jobs may still retain connections roughly in proportion to `fibers`.
+- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. This works with both `threads` and `fibers` workers as long as the supervisor is running in the default `fork` mode. **Note**: this option is ignored only when the supervisor itself is [running in `async` mode](#fork-vs-async-mode).
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 
 
@@ -365,7 +376,17 @@ queues: back*
 
 ### Threads, processes, and signals
 
-Workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling.
+By default, workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Workers can also be configured with `fibers`, in which case claimed jobs are executed as fibers on a single reactor thread and bounded by the worker's fiber count. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling.
+
+Fiber worker execution is best suited for cooperative, mostly I/O-bound jobs. Blocking or CPU-heavy work still blocks the single reactor thread, so it should not be expected to outperform thread mode for every workload.
+
+Because fiber workers run multiple fibers on a single thread, Rails must also isolate execution state per fiber rather than per thread. If your app keeps the default thread-scoped isolation level, Solid Queue will raise a boot-time error instead of running fiber workers with shared Active Record state.
+
+Keep in mind that `config.active_support.isolation_level = :fiber` applies to your whole application, not just to Solid Queue: if you run Solid Queue inside Puma via [the plugin](#puma-plugin), or combine fiber workers with thread workers in the same process using the supervisor's `async` mode, everything in that process will use fiber-scoped execution state. This is fully supported by Rails, but it's a global setting worth being deliberate about.
+
+On Rails 7.2 and later, fiber workers can often use a much smaller queue database pool than an equivalent thread pool. A practical starting point is `3-5` queue database connections per worker process: one for job execution, one for polling, one for heartbeats, plus some headroom. In the default `fork` supervisor mode, that guidance applies per worker process. In supervisor `async` mode, all workers share one process, so add together the requirements for the workers running there.
+
+That lower-pool guidance depends on job code not holding connections open across non-blocking waits. APIs such as `ActiveRecord::Base.connection`, `lease_connection`, `connection_pool.checkout`, or long-lived `with_connection` / transaction blocks can pin connections and push fiber workers back toward thread-like pool usage. On Rails 7.1, plan conservatively and assume the configured fiber count can still grow queue database connection usage.
 
 The supervisor is in charge of managing these processes, and it responds to the following signals when running in its own process via `bin/jobs` or with [the Puma plugin](#puma-plugin) with the default `fork` mode:
 - `TERM`, `INT`: starts graceful termination. The supervisor will send a `TERM` signal to its supervised processes, and it'll wait up to `SolidQueue.shutdown_timeout` time until they're done. If any supervised processes are still around by then, it'll send a `QUIT` signal to them to indicate they must exit.
@@ -377,6 +398,10 @@ On Windows, the `QUIT` signal can't be trapped, so the supervisor only responds
 
 If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. Jobs that were claimed by processes with an expired heartbeat will be marked as failed with a `SolidQueue::Processes::ProcessPrunedError`. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
 
+Worker heartbeats are driven by a separate timer task, not by the worker execution backend itself. This means fiber workers do not rely on the reactor loop to prove liveness. However, liveness is still tracked at the worker-process level, not at the individual thread or fiber level.
+
+This means finished and failed jobs still follow the normal Solid Queue lifecycle, but a single stuck job can remain claimed if the worker process itself is still alive. If you need stronger stuck-job detection, that requires an explicit timeout or watchdog mechanism on top of process heartbeats.
+
 In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.
 
 
@@ -392,7 +417,7 @@ _Note_: The settings in this section should be set in your `config/application.r
 
 There are several settings that control how Solid Queue works that you can set as well:
 - `logger`: the logger you want Solid Queue to use. Defaults to the app logger.
-- `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap asynchronous operations, defaults to the app executor
+- `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap background operations, defaults to the app executor
 - `on_thread_error`: custom lambda/Proc to call when there's an error within a Solid Queue thread that takes the exception raised as argument. Defaults to
 
   ```ruby
@@ -404,6 +429,7 @@ There are several settings that control how Solid Queue works that you can set a
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8; for MariaDB, versions < 10.6; and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
+- `fork_boot_timeout`: how long a forked process can take to finish booting before the supervisor replaces it—defaults to 5 minutes. It only applies in the default `fork` mode.
 - `shutdown_timeout`: time the supervisor will wait since it sent the `TERM` signal to its supervised processes before sending a `QUIT` version to them requesting immediate termination—defaults to 5 seconds.
 - `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to `true`.
 - `supervisor_pidfile`: path to a pidfile that the supervisor will create when booting to prevent running more than one supervisor in the same host, or in case you want to use it for a health check. It's `nil` by default.
@@ -709,6 +735,8 @@ bin/jobs --recurring_schedule_file=config/schedule.yml
 
 You can completely disable recurring tasks by setting the environment variable `SOLID_QUEUE_SKIP_RECURRING=true` or by using the `--skip-recurring` option with `bin/jobs`.
 
+To run only the scheduler (no workers or dispatchers), set `SOLID_QUEUE_ONLY_RECURRING=true` or use `--only-recurring` with `bin/jobs`.
+
 The configuration itself looks like this:
 
 ```yml
@@ -802,6 +830,8 @@ SolidQueue.unschedule_recurring_task("my_dynamic_task")
 
 Only dynamic tasks can be unscheduled at runtime. Attempting to unschedule a static task (defined in `config/recurring.yml`) will raise an `ActiveRecord::RecordNotFound` error.
 
+To update an existing dynamic task, unschedule it and then schedule it again with the new options. A running scheduler only detects dynamic tasks being created and deleted, so updating a `SolidQueue::RecurringTask` record in place (for example, changing its `schedule` with `update!`) won't be picked up until the scheduler restarts.
+
 Tasks scheduled like this persist between Solid Queue's restarts and won't stop running until you manually unschedule them. 
 
 ## Inspiration

data/app/models/solid_queue/record/distinct_values.rb

@@ -19,23 +19,25 @@ module SolidQueue
 
         private
           def loose_index_scan_emulation_needed?
-            connection.adapter_name == "PostgreSQL"
+            connection_pool.with_connection { |connection| connection.adapter_name == "PostgreSQL" }
           end
 
           # Emulates a loose index scan, honoring the current scope (e.g. LIKE prefixes)
           # by building the anchor and the recursive step as scoped relations, whose
           # #to_sql inlines any bind parameters so they can be embedded in the raw CTE.
           def loose_distinct_via_recursive_cte(column)
-            col = connection.quote_column_name(column)
+            connection_pool.with_connection do |connection|
+              col = connection.quote_column_name(column)
 
-            connection.select_values(<<~SQL.squish)
-              WITH RECURSIVE t AS (
-                (#{next_distinct_value(col, "#{col} IS NOT NULL")})
-                UNION ALL
-                SELECT (#{next_distinct_value(col, "#{col} > t.#{col}")}) FROM t WHERE t.#{col} IS NOT NULL
-              )
-              SELECT #{col} FROM t WHERE #{col} IS NOT NULL
-            SQL
+              connection.select_values(<<~SQL.squish)
+                WITH RECURSIVE t AS (
+                  (#{next_distinct_value(col, "#{col} IS NOT NULL")})
+                  UNION ALL
+                  SELECT (#{next_distinct_value(col, "#{col} > t.#{col}")}) FROM t WHERE t.#{col} IS NOT NULL
+                )
+                SELECT #{col} FROM t WHERE #{col} IS NOT NULL
+              SQL
+            end
           end
 
           # Smallest value of `col` within the current scope that matches `condition`.

data/app/models/solid_queue/record.rb

@@ -24,6 +24,16 @@ module SolidQueue
         end
       end
 
+      def warn_about_pending_migrations
+        SolidQueue.deprecator.warn(<<~DEPRECATION)
+          Solid Queue has pending database migrations. To get the new migration files, run:
+            rails solid_queue:update
+          And then:
+            rails db:migrate
+          These migrations will be required after version #{SolidQueue.next_major_version}.0
+        DEPRECATION
+      end
+
       # Pass index hints to the query optimizer using SQL comment hints.
       # Uses MySQL 8 optimizer hint query comments, which SQLite and
       # PostgreSQL ignore.

data/lib/generators/solid_queue/update/update_generator.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+require "rails/generators/active_record"
+
+class SolidQueue::UpdateGenerator < Rails::Generators::Base
+  include ActiveRecord::Generators::Migration
+
+  source_root File.expand_path("templates", __dir__)
+
+  class_option :database, type: :string, aliases: %i[ --db ], default: "queue",
+    desc: "The database that Solid Queue uses. Defaults to `queue`"
+
+  def copy_new_migrations
+    Dir.glob(File.join(self.class.source_root, "db", "*.rb")).each do |migration_file|
+      name = File.basename(migration_file)
+      migration_template File.join("db", name), File.join(db_migrate_path, name), skip: true
+    end
+  end
+end

data/lib/solid_queue/cli.rb

@@ -20,6 +20,10 @@ module SolidQueue
       desc: "Whether to skip recurring tasks scheduling",
       banner: "SOLID_QUEUE_SKIP_RECURRING"
 
+    class_option :only_recurring, type: :boolean,
+      desc: "Whether to run only the scheduler process for recurring tasks",
+      banner: "SOLID_QUEUE_ONLY_RECURRING"
+
     def self.exit_on_failure?
       true
     end

data/lib/solid_queue/configuration.rb

@@ -6,7 +6,9 @@ module SolidQueue
     include ActiveModel::Validations::Callbacks
 
     validate :ensure_configured_processes, :ensure_valid_recurring_tasks
-    validate :warn_about_incorrectly_sized_thread_pool, :warn_about_missing_config_files
+    validate :ensure_valid_worker_execution_options
+    validate :ensure_fiber_workers_have_required_dependency, :ensure_fiber_workers_use_supported_isolation_level
+    validate :warn_about_incorrectly_sized_database_pool, :warn_about_missing_config_files
 
     before_validation { warnings.clear }
 
@@ -37,13 +39,17 @@ module SolidQueue
 
     DEFAULT_CONFIG_FILE_PATH = "config/queue.yml"
     DEFAULT_RECURRING_SCHEDULE_FILE_PATH = "config/recurring.yml"
+    FIBER_QUERY_SCOPED_CONNECTIONS_VERSION = Gem::Version.new("7.2.0")
 
     def initialize(**options)
       @options = options.with_defaults(default_options)
     end
 
     def configured_processes
-      if only_work? then workers
+      if only_work?
+        workers
+      elsif only_recurring?
+        schedulers
       else
         dispatchers + workers + schedulers
       end
@@ -96,12 +102,12 @@ module SolidQueue
         end
       end
 
-      def warn_about_incorrectly_sized_thread_pool
+      def warn_about_incorrectly_sized_database_pool
         db_pool_size = SolidQueue::Record.connection_pool&.size
 
-        if db_pool_size && db_pool_size < estimated_number_of_threads
-          warnings.add(:base, "Warning: Solid Queue is configured to use #{estimated_number_of_threads} threads but the " \
-            "database connection pool is #{db_pool_size}. Increase it in `config/database.yml`")
+        if db_pool_size && db_pool_size < estimated_database_pool_size
+          warnings.add(:base, "Warning: Solid Queue needs at least #{estimated_database_pool_size} database connections " \
+            "for the configured workers but the database connection pool is #{db_pool_size}. Increase it in `config/database.yml`")
         end
       rescue ActiveRecord::ActiveRecordError
         # No usable database connection. Skip the pool-size warning in that case.
@@ -118,6 +124,34 @@ module SolidQueue
         end
       end
 
+      def ensure_valid_worker_execution_options
+        workers_options.each do |options|
+          if options.key?(:threads) && options.key?(:fibers)
+            errors.add(:base, "Workers can specify either `threads` or `fibers`, but not both.")
+          end
+        end
+      end
+
+      def ensure_fiber_workers_have_required_dependency
+        return unless workers_options.any? { |options| fiber_worker?(options) }
+
+        require "async"
+        require "async/semaphore"
+      rescue LoadError
+        errors.add(:base, "Fiber workers require the `async` gem. " \
+          "Add `gem \"async\"` to your Gemfile to configure workers with `fibers`.")
+      end
+
+      def ensure_fiber_workers_use_supported_isolation_level
+        return unless workers_options.any? { |options| fiber_worker?(options) }
+
+        unless ActiveSupport::IsolatedExecutionState.isolation_level == :fiber
+          errors.add(:base, "Fiber workers require fiber-scoped isolated execution state. " \
+            "Set `config.active_support.isolation_level = :fiber` in your Rails configuration " \
+            "(or `ActiveSupport::IsolatedExecutionState.isolation_level = :fiber` outside Rails).")
+        end
+      end
+
       def default_options
         {
           mode: ENV["SOLID_QUEUE_SUPERVISOR_MODE"] || :fork,
@@ -126,6 +160,7 @@ module SolidQueue
           recurring_schedule_file: Rails.root.join(ENV["SOLID_QUEUE_RECURRING_SCHEDULE"] || DEFAULT_RECURRING_SCHEDULE_FILE_PATH),
           only_work: false,
           only_dispatch: false,
+          only_recurring: ActiveModel::Type::Boolean.new.cast(ENV["SOLID_QUEUE_ONLY_RECURRING"]),
           skip_recurring: ActiveModel::Type::Boolean.new.cast(ENV["SOLID_QUEUE_SKIP_RECURRING"])
         }
       end
@@ -142,6 +177,10 @@ module SolidQueue
         options[:only_dispatch]
       end
 
+      def only_recurring?
+        options[:only_recurring]
+      end
+
       def skip_recurring_tasks?
         options[:skip_recurring] || only_work?
       end
@@ -154,7 +193,8 @@ module SolidQueue
             1
           end
 
-          processes.times.map { Process.new(:worker, worker_options.with_defaults(WORKER_DEFAULTS)) }
+          defaults = worker_defaults_for(worker_options)
+          processes.times.map { Process.new(:worker, worker_options.with_defaults(defaults)) }
         end
       end
 
@@ -248,10 +288,42 @@ module SolidQueue
         end
       end
 
-      def estimated_number_of_threads
-        # At most "threads" in each worker + 1 thread for the worker + 1 thread for the heartbeat task
-        thread_count = workers_options.map { |options| options.fetch(:threads, WORKER_DEFAULTS[:threads]) }.max
-        (thread_count || 1) + 2
+      def estimated_database_pool_size
+        worker_pool_size = workers_options.map { |options| estimated_database_pool_size_for_worker(options) }.max
+        worker_pool_size || 1
+      end
+
+      def estimated_database_pool_size_for_worker(options)
+        # Connections used to execute jobs + 1 for the worker's polling thread + 1 for the heartbeat task
+        estimated_execution_connections_for_worker(options) + 2
+      end
+
+      def worker_capacity(options)
+        options[:fibers] || options[:threads] || WORKER_DEFAULTS[:threads]
+      end
+
+      def estimated_execution_connections_for_worker(options)
+        fiber_worker?(options) ? fiber_execution_connections_for_worker(options) : worker_capacity(options)
+      end
+
+      def fiber_execution_connections_for_worker(options)
+        fiber_jobs_release_connections_between_queries? ? 1 : worker_capacity(options)
+      end
+
+      def fiber_jobs_release_connections_between_queries?
+        ActiveRecord.gem_version >= FIBER_QUERY_SCOPED_CONNECTIONS_VERSION
+      end
+
+      def fiber_worker?(options)
+        options.key?(:fibers)
+      end
+
+      def worker_defaults_for(options)
+        if fiber_worker?(options)
+          WORKER_DEFAULTS.except(:threads)
+        else
+          WORKER_DEFAULTS
+        end
       end
   end
 end

data/lib/solid_queue/engine.rb

@@ -43,5 +43,9 @@ module SolidQueue
         include ActiveJob::ConcurrencyControls
       end
     end
+
+    initializer "solid_queue.deprecator" do |app|
+      app.deprecators[:solid_queue] = SolidQueue.deprecator
+    end
   end
 end

data/lib/solid_queue/fiber_pool.rb

@@ -0,0 +1,130 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class FiberPool < Pool
+    def initialize(size, on_idle: nil)
+      super
+
+      @state_mutex = Mutex.new
+      @shutdown = false
+      @fatal_error = nil
+      @boot_queue = Thread::Queue.new
+      @pending_executions = Thread::Queue.new
+      @reactor_thread = nil
+    end
+
+    def post(execution)
+      raise_if_fatal_error!
+      raise RuntimeError, "Execution pool is shutting down" if shutdown?
+
+      super
+    end
+
+    def available_capacity
+      raise_if_fatal_error!
+      super
+    end
+
+    def shutdown
+      state_mutex.synchronize do
+        next false if @shutdown
+
+        @shutdown = true
+      end.tap do |shut_down|
+        # Wake the reactor: already-queued executions are drained before the
+        # blocked pop in +wait_for_executions+ returns nil
+        pending_executions.close if shut_down
+      end
+    end
+
+    def shutdown?
+      state_mutex.synchronize { @shutdown }
+    end
+
+    def wait_for_termination(timeout)
+      reactor_thread&.join(timeout)
+    end
+
+    private
+      attr_reader :boot_queue, :pending_executions, :reactor_thread, :state_mutex
+
+      def name
+        @name ||= "solid_queue-fiber-pool-#{object_id}"
+      end
+
+      def schedule(execution)
+        start_reactor_if_needed
+        pending_executions << execution
+      end
+
+      # The reactor thread is started lazily, when the first execution is posted,
+      # so that the pool can be safely built before forking: in the default fork
+      # supervisor mode, workers are instantiated in the supervisor process, and
+      # a thread started there wouldn't survive the fork. The async gem is also
+      # required lazily here, so that setups without fiber workers never load it.
+      def start_reactor_if_needed
+        @reactor_thread ||= begin
+          require "async"
+          require "async/semaphore"
+
+          start_reactor.tap do
+            boot_result = boot_queue.pop
+            raise boot_result if boot_result.is_a?(Exception)
+          end
+        end
+      end
+
+      def start_reactor
+        create_thread do
+          Async do |task|
+            semaphore = Async::Semaphore.new(size, parent: task)
+            boot_queue << :ready
+
+            # The reactor exits when all in-flight execution fibers, children
+            # of this task, have finished
+            wait_for_executions(semaphore)
+          end
+        rescue Exception => error
+          register_fatal_error(error)
+          raise
+        end
+      end
+
+      def wait_for_executions(semaphore)
+        # Thread::Queue#pop is fiber-scheduler-aware: it suspends this fiber, letting
+        # execution fibers run, and wakes the reactor when the poller thread pushes new
+        # work or closes the queue on shutdown, after which it drains any remaining
+        # executions and returns nil
+        while execution = pending_executions.pop
+          semaphore.async(execution) do |_execution_task, scheduled_execution|
+            perform_execution(scheduled_execution)
+          end
+        end
+      end
+
+      def perform_execution(execution)
+        wrap_in_app_executor { execution.perform }
+      rescue Async::Stop => error
+        handle_thread_error(error)
+        register_fatal_error(error)
+      rescue Exception => error
+        handle_thread_error(error)
+      ensure
+        restore_capacity
+      end
+
+      def register_fatal_error(error)
+        state_mutex.synchronize do
+          @fatal_error ||= error
+        end
+
+        boot_queue << error if boot_queue.empty?
+        on_idle&.call
+      end
+
+      def raise_if_fatal_error!
+        error = state_mutex.synchronize { @fatal_error }
+        raise error if error
+      end
+  end
+end

data/lib/solid_queue/fork_supervisor.rb

@@ -31,6 +31,21 @@ module SolidQueue
 
         replace_fork(pid, status)
       end
+
+      check_boot_timeouts
+    end
+
+    def check_boot_timeouts
+      process_instances.each do |pid, instance|
+        terminate_unready_process(pid) if instance.boot_timed_out?
+      end
+    end
+
+    def terminate_unready_process(pid)
+      SolidQueue.instrument(:fork_boot_timeout, process: process_instances[pid], pid: pid) do
+        # A child stuck in boot cannot reach its run loop to stop gracefully
+        signal_process(pid, :KILL)
+      end
     end
 
     def reap_terminated_forks
@@ -38,9 +53,13 @@ module SolidQueue
         pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
         break unless pid
 
-        if (terminated_fork = process_instances.delete(pid)) && (!status.exited? || status.exitstatus.to_i > 0)
-          error = Processes::ProcessExitError.new(status)
-          release_claimed_jobs_by(terminated_fork, with_error: error)
+        if terminated_fork = process_instances.delete(pid)
+          terminated_fork.mark_as_reaped
+
+          if !status.exited? || status.exitstatus.to_i > 0
+            error = Processes::ProcessExitError.new(status)
+            release_claimed_jobs_by(terminated_fork, with_error: error)
+          end
         end
 
         configured_processes.delete(pid)
@@ -52,6 +71,7 @@ module SolidQueue
     def replace_fork(pid, status)
       SolidQueue.instrument(:replace_fork, supervisor_pid: ::Process.pid, pid: pid, status: status) do |payload|
         if terminated_fork = process_instances.delete(pid)
+          terminated_fork.mark_as_reaped
           payload[:fork] = terminated_fork
           error = Processes::ProcessExitError.new(status)
           release_claimed_jobs_by(terminated_fork, with_error: error)

data/lib/solid_queue/log_subscriber.rb

@@ -161,6 +161,11 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
     end
   end
 
+  def fork_boot_timeout(event)
+    process = event.payload[:process]
+    warn formatted_event(event, action: "Terminate #{process.kind} that failed to boot in time", **event.payload.slice(:pid).merge(hostname: process.hostname, name: process.name))
+  end
+
   private
     def formatted_event(event, action:, **attributes)
       "SolidQueue-#{SolidQueue::VERSION} #{action} (#{event.duration.round(1)}ms)  #{formatted_attributes(**attributes)}"

data/lib/solid_queue/pool.rb

@@ -4,51 +4,72 @@ module SolidQueue
   class Pool
     include AppExecutor
 
-    attr_reader :size
+    def self.build(type:, size:, on_idle: nil)
+      SolidQueue.const_get("#{type.to_s.camelize}Pool").new(size, on_idle: on_idle)
+    end
 
-    delegate :shutdown, :shutdown?, :wait_for_termination, to: :executor
+    attr_reader :size
 
     def initialize(size, on_idle: nil)
       @size = size
       @on_idle = on_idle
-      @available_threads = Concurrent::AtomicFixnum.new(size)
+      @available_capacity = size
       @mutex = Mutex.new
     end
 
+    def type
+      self.class.name.demodulize.delete_suffix("Pool").underscore.to_sym
+    end
+
     def post(execution)
-      available_threads.decrement
-
-      Concurrent::Promises.future_on(executor, execution) do |thread_execution|
-        wrap_in_app_executor do
-          thread_execution.perform
-        ensure
-          available_threads.increment
-          mutex.synchronize { on_idle.try(:call) if idle? }
-        end
-      end.on_rejection! do |e|
-        handle_thread_error(e)
+      reserve_capacity!
+
+      begin
+        schedule(execution)
+      rescue Exception
+        restore_capacity
+        raise
       end
     end
 
-    def idle_threads
-      available_threads.value
+    def available_capacity
+      mutex.synchronize { @available_capacity }
     end
 
     def idle?
-      idle_threads > 0
+      available_capacity.positive?
     end
 
     private
-      attr_reader :available_threads, :on_idle, :mutex
+      attr_reader :mutex, :on_idle
+
+      def schedule(execution)
+        raise NotImplementedError
+      end
+
+      def perform_execution(execution)
+        wrap_in_app_executor { execution.perform }
+      rescue Exception => error
+        handle_thread_error(error)
+      ensure
+        restore_capacity
+      end
+
+      def reserve_capacity!
+        mutex.synchronize do
+          raise RuntimeError, "Execution pool is at capacity" if @available_capacity <= 0
 
-      DEFAULT_OPTIONS = {
-        min_threads: 0,
-        idletime: 60,
-        fallback_policy: :abort
-      }
+          @available_capacity -= 1
+        end
+      end
+
+      def restore_capacity
+        should_notify = mutex.synchronize do
+          @available_capacity += 1
+          @available_capacity.positive?
+        end
 
-      def executor
-        @executor ||= Concurrent::ThreadPoolExecutor.new DEFAULT_OPTIONS.merge(max_threads: size, max_queue: size)
+        on_idle&.call if should_notify
       end
   end
 end

data/lib/solid_queue/processes/runnable.rb

@@ -31,6 +31,14 @@ module SolidQueue::Processes
       !running_async? || @thread&.alive?
     end
 
+    def boot_timed_out?
+      @boot_guard.timed_out?
+    end
+
+    def mark_as_reaped
+      @boot_guard.close
+    end
+
     private
       DEFAULT_MODE = :async
 
@@ -41,11 +49,14 @@ module SolidQueue::Processes
       def run_in_mode(&block)
         case
         when running_as_fork?
-          fork(&block)
+          @boot_guard = BootGuards::ForkGuard.new
+          fork(&block).tap { @boot_guard.start }
         when running_async?
+          @boot_guard = BootGuards::NullGuard.new
           @thread = create_thread(&block)
           @thread.object_id
         else
+          @boot_guard = BootGuards::NullGuard.new
           block.call
         end
       end
@@ -59,6 +70,8 @@ module SolidQueue::Processes
             end
           end
         end
+
+        @boot_guard.complete
       end
 
       def shutting_down?
@@ -95,4 +108,76 @@ module SolidQueue::Processes
         mode.fork?
       end
   end
+
+  module BootGuards
+    # Tracks a process that shares memory with its supervisor, whose boot time
+    # doesn't need monitoring.
+    class NullGuard
+      def complete
+        @completed = true
+      end
+
+      def start
+      end
+
+      def completed?
+        @completed
+      end
+
+      def timed_out?
+        false
+      end
+
+      def close
+      end
+    end
+
+    # Tracks a forked process from the moment it's started until its boot
+    # callbacks finish, over a pipe that survives forking: the forked process
+    # writes to it when it's done booting, and its supervisor reads from it to
+    # decide whether the process is taking too long to boot and needs replacing.
+    class ForkGuard
+      def initialize
+        @reader, @writer = IO.pipe
+        @created_at = SolidQueue::Timer.monotonic_time_now
+      end
+
+      # Runs in the forked process when it has finished booting
+      def complete
+        reader.close
+        writer.write(".")
+      rescue Errno::EPIPE
+        # The supervisor stopped waiting while this process finished booting
+      ensure
+        writer.close
+      end
+
+      # Runs in the parent process right after forking
+      def start
+        writer.close
+      end
+
+      # A byte means boot completed; EOF means the process exited before
+      # finishing its boot, and will be replaced when it's reaped
+      def completed?
+        @completed ||= begin
+          completed = reader.read_nonblock(1, exception: false) != :wait_readable
+          reader.close if completed
+          completed
+        end
+      end
+
+      def timed_out?
+        !completed? && SolidQueue::Timer.monotonic_time_now - created_at >= SolidQueue.fork_boot_timeout
+      end
+
+      def close
+        reader.close unless reader.closed?
+        writer.close unless writer.closed?
+      end
+
+      private
+        attr_reader :reader, :writer, :created_at
+    end
+  end
 end

data/lib/solid_queue/tasks.rb

@@ -4,6 +4,11 @@ namespace :solid_queue do
     Rails::Command.invoke :generate, [ "solid_queue:install" ]
   end
 
+  desc "Copy any new Solid Queue migrations to the application"
+  task :update do
+    Rails::Command.invoke :generate, [ "solid_queue:update" ]
+  end
+
   desc "start solid_queue supervisor to dispatch and process jobs"
   task start: :environment do
     SolidQueue::Supervisor.start

data/lib/solid_queue/thread_pool.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class ThreadPool < Pool
+    delegate :shutdown, :shutdown?, :wait_for_termination, to: :executor
+
+    private
+      DEFAULT_OPTIONS = {
+        min_threads: 0,
+        idletime: 60,
+        fallback_policy: :abort
+      }
+
+      def schedule(execution)
+        Concurrent::Promises.future_on(executor, execution) do |thread_execution|
+          perform_execution(thread_execution)
+        end.on_rejection! do |error|
+          # Backstop for errors raised outside perform_execution's own rescue,
+          # such as when restoring capacity or waking up the worker
+          handle_thread_error(error)
+        end
+      end
+
+      def executor
+        @executor ||= Concurrent::ThreadPoolExecutor.new DEFAULT_OPTIONS.merge(max_threads: size, max_queue: size)
+      end
+  end
+end

data/lib/solid_queue/timer.rb

@@ -20,9 +20,8 @@ module SolidQueue
       end
     end
 
-    private
-      def monotonic_time_now
-        ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
-      end
+    def monotonic_time_now
+      ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+    end
   end
 end

data/lib/solid_queue/version.rb

@@ -1,3 +1,7 @@
 module SolidQueue
-  VERSION = "1.5.0"
+  VERSION = "1.6.0"
+
+  def self.next_major_version
+    Gem::Version.new(VERSION).segments.first + 1
+  end
 end

data/lib/solid_queue/worker.rb

@@ -11,18 +11,24 @@ module SolidQueue
     attr_reader :queues, :pool
 
     def initialize(**options)
+      execution_pool_type = options.key?(:fibers) ? :fiber : :thread
+
       options = options.dup.with_defaults(SolidQueue::Configuration::WORKER_DEFAULTS)
+      execution_pool_size = execution_pool_type == :fiber ? options[:fibers] : options[:threads]
 
       # Ensure that the queues array is deep frozen to prevent accidental modification
       @queues = Array(options[:queues]).map(&:freeze).freeze
 
-      @pool = Pool.new(options[:threads], on_idle: -> { wake_up })
+      @pool = Pool.build \
+        type: execution_pool_type,
+        size: execution_pool_size,
+        on_idle: -> { wake_up }
 
       super(**options)
     end
 
     def metadata
-      super.merge(queues: queues.join(","), thread_pool_size: pool.size)
+      super.merge(queues: queues.join(","), pool_type: pool.type, pool_size: pool.size)
     end
 
     private
@@ -38,7 +44,7 @@ module SolidQueue
 
       def claim_executions
         with_polling_volume do
-          SolidQueue::ReadyExecution.claim(queues, pool.idle_threads, process_id)
+          SolidQueue::ReadyExecution.claim(queues, pool.available_capacity, process_id)
         end
       end
 

data/lib/solid_queue.rb

@@ -29,6 +29,7 @@ module SolidQueue
 
   mattr_accessor :process_heartbeat_interval, default: 60.seconds
   mattr_accessor :process_alive_threshold, default: 5.minutes
+  mattr_accessor :fork_boot_timeout, default: 5.minutes
 
   mattr_accessor :shutdown_timeout, default: 5.seconds
 
@@ -86,6 +87,10 @@ module SolidQueue
     preserve_finished_jobs
   end
 
+  def deprecator
+    @deprecator ||= ActiveSupport::Deprecation.new(next_major_version, "SolidQueue")
+  end
+
   def instrument(channel, **options, &block)
     ActiveSupport::Notifications.instrument("#{channel}.solid_queue", **options, &block)
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 1.5.0
+  version: 1.6.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-23 00:00:00.000000000 Z
+date: 2026-07-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -122,6 +122,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.9'
+- !ruby/object:Gem::Dependency
+  name: async
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.24'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.24'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -309,6 +323,7 @@ files:
 - lib/generators/solid_queue/install/templates/config/queue.yml
 - lib/generators/solid_queue/install/templates/config/recurring.yml
 - lib/generators/solid_queue/install/templates/db/queue_schema.rb
+- lib/generators/solid_queue/update/update_generator.rb
 - lib/puma/plugin/solid_queue.rb
 - lib/solid_queue.rb
 - lib/solid_queue/app_executor.rb
@@ -318,6 +333,7 @@ files:
 - lib/solid_queue/dispatcher.rb
 - lib/solid_queue/dispatcher/concurrency_maintenance.rb
 - lib/solid_queue/engine.rb
+- lib/solid_queue/fiber_pool.rb
 - lib/solid_queue/fork_supervisor.rb
 - lib/solid_queue/lifecycle_hooks.rb
 - lib/solid_queue/log_subscriber.rb
@@ -342,6 +358,7 @@ files:
 - lib/solid_queue/supervisor/pidfiled.rb
 - lib/solid_queue/supervisor/signals.rb
 - lib/solid_queue/tasks.rb
+- lib/solid_queue/thread_pool.rb
 - lib/solid_queue/timer.rb
 - lib/solid_queue/version.rb
 - lib/solid_queue/worker.rb