solid_queue 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1cb50c543011419d9ad0cee9bbf9009088732dd3025b1c03fb89c9155e81b4ee
-  data.tar.gz: 97ab2ed127edf72e5cea4ac78f69778f1728b84ef4ec4922dc5fc885ffa123c2
+  metadata.gz: 72d9dc2e30127c7b1fe101648d4d18b04af2d33491c729816e25dcf488246b54
+  data.tar.gz: e39dcad251d4124355adcedfa9cb1ee88eea12043eb8505568332d9fcacee428
 SHA512:
-  metadata.gz: eb2150d9025eba950e409d9b84543cc41e019f997dfb4c65c4d7819365576198ddf0253847c7937594eb6930a0b528415c68456c73f2afa084353ff72fdc5511
-  data.tar.gz: 4918c8b1bc0a37ef6b888408ba7fdfdb96401cb5515d24138eb850d726a17913e4544e53c507c6069eaf1f905b6fb5c9e9406877b870eec135d0f62141480962
+  metadata.gz: 967732a5e679269543869ca64aa410bfc43c935ad3fb6856de0860bbe5cee58716e389b5935e48a91c6c658a84fa79aff9f5eae93ba3931344d31aff0d45c8fc
+  data.tar.gz: 523af9f1b26a137d99ade0805bdc955cb81584971308928d5e16e83d3c46861947deaaa95ff96775d7eceb913f3d8995eca14707cda4533f7c3ee39f20afb650

data/README.md

@@ -2,27 +2,11 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon._
 
-## Usage
-To set Solid Queue as your Active Job's queue backend, you should add this to your environment config:
-```ruby
-# config/environments/production.rb
-config.active_job.queue_adapter = :solid_queue
-```
-
-Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
-
-```ruby
-# app/jobs/my_job.rb
-
-class MyJob < ApplicationJob
-  self.queue_adapter = :solid_queue
-  # ...
-end
-```
+Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
-## Installation
+## Installation and usage
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -39,21 +23,51 @@ Or install it yourself as:
 $ gem install solid_queue
 ```
 
-Add the migration to your app and run it:
+Now, you need to install the necessary migrations and configure the Active Job's adapter. You can do both at once using the provided generator:
+
+```bash
+$ bin/rails generate solid_queue:install
 ```
+
+This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
+
+Alternatively, you can add the only the migration to your app:
+```bash
 $ bin/rails solid_queue:install:migrations
-$ bin/rails db:migrate
 ```
 
-With this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+And set Solid Queue as your Active Job's queue backend manually, in your environment config:
+```ruby
+# config/environments/production.rb
+config.active_job.queue_adapter = :solid_queue
+```
+
+Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+
+```ruby
+# app/jobs/my_job.rb
+
+class MyJob < ApplicationJob
+  self.queue_adapter = :solid_queue
+  # ...
+end
+```
+
+Finally, you need to run the migrations:
+
+```bash
+$ bin/rails db:migrate
 ```
+
+After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+```bash
 $ bundle exec rake solid_queue:start
 ```
 
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
 ## Requirements
-Besides Rails 7, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
+Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
 
 ## Configuration
 
@@ -61,7 +75,7 @@ Besides Rails 7, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as the
 
 We have three types of processes in Solid Queue:
 - _Workers_ are in charge of picking jobs ready to run from queues and processing them. They work off the `solid_queue_ready_executions` table.
-- _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_jobs` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They also do some maintenance work related to concurrency controls.
+- _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They also do some maintenance work related to concurrency controls.
 - The _supervisor_ forks workers and dispatchers according to the configuration, controls their heartbeats, and sends them signals to stop and start them when needed.
 
 By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
@@ -71,11 +85,12 @@ production:
   dispatchers:
     - polling_interval: 1
       batch_size: 500
+      concurrency_maintenance_interval: 300
   workers:
-    - queues: *
+    - queues: "*"
       threads: 3
       polling_interval: 2
-    - queues: real_time,background
+    - queues: [ real_time, background ]
       threads: 5
       polling_interval: 0.1
       processes: 3
@@ -83,20 +98,23 @@ production:
 
 Everything is optional. If no configuration is provided, Solid Queue will run with one dispatcher and one worker with default settings.
 
-- `polling_interval`: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to `5` seconds for dispatchers and `1` second for workers.
-- `batch_size`: the dispatcher will dispatch jobs in batches of this size.
-- `queues`: the list of queues that workers will pick jobs from. You can use `*` to indicate all queues (which is also the default and the behaviour you'll get if you omit this). You can provide a comma-separated list of queues. Jobs will be polled from those queues in order, so for example, with `real_time,background`, no jobs will be taken from `background` unless there aren't any more jobs waiting in `real_time`. You can also provide a prefix with a wildcard to match queues starting with a prefix. For example:
-```yml
-staging:
-  workers:
-    - queues: staging*
-      threads: 3
-      polling_interval: 5
+- `polling_interval`: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to `1` second for dispatchers and `0.1` seconds for workers.
+- `batch_size`: the dispatcher will dispatch jobs in batches of this size. The default is 500.
+- `concurrency_maintenance_interval`: the time interval in seconds that the dispatcher will wait before checking for blocked jobs that can be unblocked. Read more about [concurrency controls](#concurrency-controls) to learn more about this setting. It defaults to `600` seconds.
+- `queues`: the list of queues that workers will pick jobs from. You can use `*` to indicate all queues (which is also the default and the behaviour you'll get if you omit this). You can provide a single queue, or a list of queues as an array. Jobs will be polled from those queues in order, so for example, with `[ real_time, background ]`, no jobs will be taken from `background` unless there aren't any more jobs waiting in `real_time`. You can also provide a prefix with a wildcard to match queues starting with a prefix. For example:
 
-```
-This will create a worker fetching jobs from all queues starting with `staging`. The wildcard `*` is only allowed on its own or at the end of a queue name; you can't specify queue names such as `*_some_queue`. These will be ignored.
+  ```yml
+  staging:
+    workers:
+      - queues: staging*
+        threads: 3
+        polling_interval: 5
+
+  ```
 
-Finally, you can combine prefixes with exact names, like `staging*, background`, and the behaviour with respect to order will be the same as with only exact names.
+  This will create a worker fetching jobs from all queues starting with `staging`. The wildcard `*` is only allowed on its own or at the end of a queue name; you can't specify queue names such as `*_some_queue`. These will be ignored.
+
+  Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
 - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `5`. Only workers have this setting.
 - `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
 
@@ -129,23 +147,25 @@ There are several settings that control how Solid Queue works that you can set a
 - `logger`: the logger you want Solid Queue to use. Defaults to the app logger.
 - `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap asynchronous operations, defaults to the app executor
 - `on_thread_error`: custom lambda/Proc to call when there's an error within a thread that takes the exception raised as argument. Defaults to
-```ruby
--> (exception) { Rails.error.report(exception, handled: false) }
-```
+
+  ```ruby
+  -> (exception) { Rails.error.report(exception, handled: false) }
+  ```
 - `connects_to`: a custom database configuration that will be used in the abstract `SolidQueue::Record` Active Record model. This is required to use a different database than the main app. For example:
-```ruby
+
+  ```ruby
   # Use a separate DB for Solid Queue
   config.solid_queue.connects_to = { database: { writing: :solid_queue_primary, reading: :solid_queue_replica } }
-```
+  ```
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
-- `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to to 60 seconds.
-- `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to to 5 minutes.
-- `shutdown_timeout`: time the supervisor will wait since it sent the `TERM` signal to its supervised processes before sending a `QUIT` version to them requesting immediate termination—defaults to to 5 seconds.
-- `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to to `false`.
+- `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
+- `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
+- `shutdown_timeout`: time the supervisor will wait since it sent the `TERM` signal to its supervised processes before sending a `QUIT` version to them requesting immediate termination—defaults to 5 seconds.
+- `silence_polling`: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to `false`.
 - `supervisor_pidfile`: path to a pidfile that the supervisor will create when booting to prevent running more than one supervisor in the same host, or in case you want to use it for a health check. It's `nil` by default.
-- `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to to `true`.
-- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, but this will happen automatically in the near future.
-- `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to to 3 minutes.
+- `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to `true`.
+- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, but this will happen automatically in the near future.
+- `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
 
 
 ## Concurrency controls
@@ -197,6 +217,20 @@ Note that the `duration` setting depends indirectly on the value for `concurrenc
 
 Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for gaining the lock, and whenever they get it, they'll be run. It doesn't matter if they had gained the lock already in the past.
 
+## Failed jobs and retries
+
+Solid Queue doesn't include any automatic retry mechanism, it [relies on Active Job for this](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs). Jobs that fail will be kept in the system, and a _failed execution_ (a record in the `solid_queue_failed_executions` table) will be created for these. The job will stay there until manually discarded or re-enqueued. You can do this in a console as:
+```ruby
+failed_execution = SolidQueue::FailedExecution.find(...) # Find the failed execution related to your job
+failed_execution.error # inspect the error
+
+failed_execution.retry # This will re-enqueue the job as if it was enqueued for the first time
+failed_execution.discard # This will delete the job from the system
+```
+
+We're planning to release a dashboard called _Mission Control_, where, among other things, you'll be able to examine and retry/discard failed jobs, one by one, or in bulk.
+
+
 ## Puma plugin
 We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add
 ```ruby
@@ -211,20 +245,21 @@ to your `puma.rb` configuration.
 If you prefer not to rely on this, or avoid relying on it unintentionally, you should make sure that:
 - Your jobs relying on specific records are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
 - Or, to opt out completely from this behaviour, configure a database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
-```ruby
-class ApplicationRecord < ActiveRecord::Base
-  self.abstract_class = true
 
-  connects_to database: { writing: :primary, reading: :replica }
-```
+  ```ruby
+  class ApplicationRecord < ActiveRecord::Base
+    self.abstract_class = true
 
-```ruby
-  solid_queue.config.connects_to { database: { writing: :primary, reading: :replica } }
-```
+    connects_to database: { writing: :primary, reading: :replica }
+  ```
+
+  ```ruby
+  config.solid_queue.connects_to = { database: { writing: :primary, reading: :replica } }
+  ```
 
 ## Inspiration
 
-Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job).
+Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.
 
 ## License
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/app/models/solid_queue/blocked_execution.rb

@@ -2,7 +2,7 @@
 
 module SolidQueue
   class BlockedExecution < Execution
-    assume_attributes_from_job :concurrency_key
+    assumes_attributes_from_job :concurrency_key
     before_create :set_expires_at
 
     has_one :semaphore, foreign_key: :key, primary_key: :concurrency_key
@@ -24,7 +24,7 @@ module SolidQueue
 
       def release_one(concurrency_key)
         transaction do
-          ordered.where(concurrency_key: concurrency_key).limit(1).lock.each(&:release)
+          ordered.where(concurrency_key: concurrency_key).limit(1).non_blocking_lock.each(&:release)
         end
       end
 

data/app/models/solid_queue/claimed_execution.rb

@@ -46,7 +46,6 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
 
   private
     def execute
-      SolidQueue.logger.info("[SolidQueue] Performing job #{job.id} - #{job.active_job_id}")
       ActiveJob::Base.execute(job.arguments)
       Result.new(true, nil)
     rescue Exception => e
@@ -58,8 +57,6 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
         job.finished!
         destroy!
       end
-
-      SolidQueue.logger.info("[SolidQueue] Performed job #{job.id} - #{job.active_job_id}")
     end
 
     def failed_with(error)
@@ -67,7 +64,5 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
         job.failed_with(error)
         destroy!
       end
-
-      SolidQueue.logger.info("[SolidQueue] Failed job #{job.id} - #{job.active_job_id}")
     end
 end

data/app/models/solid_queue/execution/job_attributes.rb

@@ -5,17 +5,24 @@ module SolidQueue
     module JobAttributes
       extend ActiveSupport::Concern
 
-      ASSUMIBLE_ATTRIBUTES_FROM_JOB = %i[ queue_name priority ]
+      included do
+        class_attribute :assumible_attributes_from_job, instance_accessor: false, default: %i[ queue_name priority ]
+      end
 
       class_methods do
-        def assume_attributes_from_job(*attributes)
-          before_create -> { assume_attributes_from_job(ASSUMIBLE_ATTRIBUTES_FROM_JOB | attributes) }
+        def assumes_attributes_from_job(*attribute_names)
+          self.assumible_attributes_from_job |= attribute_names
+          before_create -> { assume_attributes_from_job }
+        end
+
+        def attributes_from_job(job)
+          job.attributes.symbolize_keys.slice(*assumible_attributes_from_job)
         end
       end
 
       private
-        def assume_attributes_from_job(attributes)
-          attributes.each do |attribute|
+        def assume_attributes_from_job
+          self.class.assumible_attributes_from_job.each do |attribute|
             send("#{attribute}=", job.send(attribute))
           end
         end

data/app/models/solid_queue/execution.rb

@@ -11,5 +11,17 @@ module SolidQueue
     belongs_to :job
 
     alias_method :discard, :destroy
+
+    class << self
+      def create_all_from_jobs(jobs)
+        insert_all execution_data_from_jobs(jobs)
+      end
+
+      def execution_data_from_jobs(jobs)
+        jobs.collect do |job|
+          attributes_from_job(job).merge(job_id: job.id)
+        end
+      end
+    end
   end
 end

data/app/models/solid_queue/failed_execution.rb

@@ -1,11 +1,7 @@
 # frozen_string_literal: true
 
 class SolidQueue::FailedExecution < SolidQueue::Execution
-  if Gem::Version.new(Rails.version) >= Gem::Version.new("7.1")
-    serialize :error, coder: JSON
-  else
-    serialize :error, JSON
-  end
+  serialize :error, coder: JSON
 
   before_create :expand_error_details_from_exception
 

data/app/models/solid_queue/job/executable.rb

@@ -6,20 +6,56 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        include Clearable, ConcurrencyControls
+        include Clearable, ConcurrencyControls, Schedulable
 
         has_one :ready_execution, dependent: :destroy
         has_one :claimed_execution, dependent: :destroy
         has_one :failed_execution, dependent: :destroy
 
-        has_one :scheduled_execution, dependent: :destroy
-
         after_create :prepare_for_execution
 
         scope :finished, -> { where.not(finished_at: nil) }
+        scope :failed, -> { includes(:failed_execution).where.not(failed_execution: { id: nil }) }
       end
 
-      %w[ ready claimed failed scheduled ].each do |status|
+      class_methods do
+        def prepare_all_for_execution(jobs)
+          due, not_yet_due = jobs.partition(&:due?)
+          dispatch_all(due) + schedule_all(not_yet_due)
+        end
+
+        def dispatch_all(jobs)
+          with_concurrency_limits, without_concurrency_limits = jobs.partition(&:concurrency_limited?)
+
+          dispatch_all_at_once(without_concurrency_limits)
+          dispatch_all_one_by_one(with_concurrency_limits)
+
+          successfully_dispatched(jobs)
+        end
+
+        private
+          def dispatch_all_at_once(jobs)
+            ReadyExecution.create_all_from_jobs jobs
+          end
+
+          def dispatch_all_one_by_one(jobs)
+            jobs.each(&:dispatch)
+          end
+
+          def successfully_dispatched(jobs)
+            dispatched_and_ready(jobs) + dispatched_and_blocked(jobs)
+          end
+
+          def dispatched_and_ready(jobs)
+            where(id: ReadyExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+
+          def dispatched_and_blocked(jobs)
+            where(id: BlockedExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+      end
+
+      %w[ ready claimed failed ].each do |status|
         define_method("#{status}?") { public_send("#{status}_execution").present? }
       end
 
@@ -49,8 +85,8 @@ module SolidQueue
         finished_at.present?
       end
 
-      def failed_with(exception)
-        FailedExecution.create_or_find_by!(job_id: id, exception: exception)
+      def due?
+        scheduled_at.nil? || scheduled_at <= Time.current
       end
 
       def discard
@@ -66,14 +102,6 @@ module SolidQueue
       end
 
       private
-        def due?
-          scheduled_at.nil? || scheduled_at <= Time.current
-        end
-
-        def schedule
-          ScheduledExecution.create_or_find_by!(job_id: id)
-        end
-
         def ready
           ReadyExecution.create_or_find_by!(job_id: id)
         end

data/app/models/solid_queue/job/schedulable.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Job
+    module Schedulable
+      extend ActiveSupport::Concern
+
+      included do
+        has_one :scheduled_execution, dependent: :destroy
+
+        scope :scheduled, -> { where.not(finished_at: nil) }
+      end
+
+      class_methods do
+        def schedule_all(jobs)
+          schedule_all_at_once(jobs)
+          successfully_scheduled(jobs)
+        end
+
+        private
+          def schedule_all_at_once(jobs)
+            ScheduledExecution.create_all_from_jobs(jobs)
+          end
+
+          def successfully_scheduled(jobs)
+            where(id: ScheduledExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+      end
+
+      def due?
+        scheduled_at.nil? || scheduled_at <= Time.current
+      end
+
+      private
+        def schedule
+          ScheduledExecution.create_or_find_by!(job_id: id)
+        end
+    end
+  end
+end

data/app/models/solid_queue/job.rb

@@ -1,38 +1,58 @@
 # frozen_string_literal: true
 
-class SolidQueue::Job < SolidQueue::Record
-  include Executable
+module SolidQueue
+  class Job < Record
+    include Executable
 
-  if Gem::Version.new(Rails.version) >= Gem::Version.new("7.1")
     serialize :arguments, coder: JSON
-  else
-    serialize :arguments, JSON
-  end
 
-  DEFAULT_PRIORITY = 0
-  DEFAULT_QUEUE_NAME = "default"
-
-  class << self
-    def enqueue_active_job(active_job, scheduled_at: Time.current)
-      enqueue \
-        queue_name: active_job.queue_name,
-        active_job_id: active_job.job_id,
-        priority: active_job.priority,
-        scheduled_at: scheduled_at,
-        class_name: active_job.class.name,
-        arguments: active_job.serialize,
-        concurrency_key: active_job.try(:concurrency_key)
-    end
+    class << self
+      def enqueue_all(active_jobs)
+        active_jobs_by_job_id = active_jobs.index_by(&:job_id)
 
-    def enqueue(**kwargs)
-      create!(**kwargs.compact.with_defaults(defaults)).tap do
-        SolidQueue.logger.debug "[SolidQueue] Enqueued job #{kwargs}"
+        transaction do
+          jobs = create_all_from_active_jobs(active_jobs)
+          prepare_all_for_execution(jobs).tap do |enqueued_jobs|
+            enqueued_jobs.each do |enqueued_job|
+              active_jobs_by_job_id[enqueued_job.active_job_id].provider_job_id = enqueued_job.id
+            end
+          end
+        end
       end
-    end
 
-    private
-      def defaults
-        { queue_name: DEFAULT_QUEUE_NAME, priority: DEFAULT_PRIORITY }
+      def enqueue(active_job, scheduled_at: Time.current)
+        active_job.scheduled_at = scheduled_at
+
+        create_from_active_job(active_job).tap do |enqueued_job|
+          active_job.provider_job_id = enqueued_job.id
+        end
       end
+
+      private
+        DEFAULT_PRIORITY = 0
+        DEFAULT_QUEUE_NAME = "default"
+
+        def create_from_active_job(active_job)
+          create!(**attributes_from_active_job(active_job))
+        end
+
+        def create_all_from_active_jobs(active_jobs)
+          job_rows = active_jobs.map { |job| attributes_from_active_job(job) }
+          insert_all(job_rows)
+          where(active_job_id: active_jobs.map(&:job_id))
+        end
+
+        def attributes_from_active_job(active_job)
+          {
+            queue_name: active_job.queue_name || DEFAULT_QUEUE_NAME,
+            active_job_id: active_job.job_id,
+            priority: active_job.priority || DEFAULT_PRIORITY,
+            scheduled_at: active_job.scheduled_at,
+            class_name: active_job.class.name,
+            arguments: active_job.serialize,
+            concurrency_key: active_job.concurrency_key
+          }
+        end
+    end
   end
 end

data/app/models/solid_queue/process/prunable.rb

@@ -9,7 +9,7 @@ module SolidQueue::Process::Prunable
 
   class_methods do
     def prune
-      prunable.lock.find_in_batches(batch_size: 50) do |batch|
+      prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
         batch.each do |process|
           SolidQueue.logger.info("[SolidQueue] Pruning dead process #{process.id} - #{process.metadata}")
           process.deregister

data/app/models/solid_queue/ready_execution.rb

@@ -4,7 +4,7 @@ module SolidQueue
   class ReadyExecution < Execution
     scope :queued_as, ->(queue_name) { where(queue_name: queue_name) }
 
-    assume_attributes_from_job
+    assumes_attributes_from_job
 
     class << self
       def claim(queue_list, limit, process_id)
@@ -26,7 +26,7 @@ module SolidQueue
         end
 
         def select_candidates(queue_relation, limit)
-          queue_relation.ordered.limit(limit).lock.pluck(:job_id)
+          queue_relation.ordered.limit(limit).non_blocking_lock.pluck(:job_id)
         end
 
         def lock_candidates(job_ids, process_id)

data/app/models/solid_queue/record.rb

@@ -6,11 +6,11 @@ module SolidQueue
 
     connects_to **SolidQueue.connects_to if SolidQueue.connects_to
 
-    def self.lock(...)
+    def self.non_blocking_lock
       if SolidQueue.use_skip_locked
-        super(Arel.sql("FOR UPDATE SKIP LOCKED"))
+        lock(Arel.sql("FOR UPDATE SKIP LOCKED"))
       else
-        super
+        lock
       end
     end
   end

data/app/models/solid_queue/scheduled_execution.rb

@@ -6,12 +6,12 @@ module SolidQueue
     scope :ordered, -> { order(scheduled_at: :asc, priority: :asc) }
     scope :next_batch, ->(batch_size) { due.ordered.limit(batch_size) }
 
-    assume_attributes_from_job :scheduled_at
+    assumes_attributes_from_job :scheduled_at
 
     class << self
       def dispatch_next_batch(batch_size)
         transaction do
-          job_ids = next_batch(batch_size).lock.pluck(:job_id)
+          job_ids = next_batch(batch_size).non_blocking_lock.pluck(:job_id)
           if job_ids.empty? then []
           else
             dispatch_batch(job_ids)
@@ -22,44 +22,12 @@ module SolidQueue
       private
         def dispatch_batch(job_ids)
           jobs = Job.where(id: job_ids)
-          with_concurrency_limits, without_concurrency_limits = jobs.partition(&:concurrency_limited?)
 
-          dispatch_at_once(without_concurrency_limits)
-          dispatch_one_by_one(with_concurrency_limits)
-
-          successfully_dispatched(job_ids).tap do |dispatched_job_ids|
+          Job.dispatch_all(jobs).map(&:id).tap do |dispatched_job_ids|
             where(job_id: dispatched_job_ids).delete_all
             SolidQueue.logger.info("[SolidQueue] Dispatched scheduled batch with #{dispatched_job_ids.size} jobs")
           end
         end
-
-        def dispatch_at_once(jobs)
-          ReadyExecution.insert_all ready_rows_from_batch(jobs)
-        end
-
-        def dispatch_one_by_one(jobs)
-          jobs.each(&:dispatch)
-        end
-
-        def ready_rows_from_batch(jobs)
-          prepared_at = Time.current
-
-          jobs.map do |job|
-            { job_id: job.id, queue_name: job.queue_name, priority: job.priority, created_at: prepared_at }
-          end
-        end
-
-        def successfully_dispatched(job_ids)
-          dispatched_and_ready(job_ids) + dispatched_and_blocked(job_ids)
-        end
-
-        def dispatched_and_ready(job_ids)
-          ReadyExecution.where(job_id: job_ids).pluck(:job_id)
-        end
-
-        def dispatched_and_blocked(job_ids)
-          BlockedExecution.where(job_id: job_ids).pluck(:job_id)
-        end
     end
   end
 end

data/db/migrate/20231211200639_create_solid_queue_tables.rb

@@ -1,4 +1,4 @@
-class CreateSolidQueueTables < ActiveRecord::Migration[7.1]
+class CreateSolidQueueTables < ActiveRecord::Migration[7.0]
   def change
     create_table :solid_queue_jobs do |t|
       t.string :queue_name, null: false

data/lib/active_job/concurrency_controls.rb

@@ -36,6 +36,10 @@ module ActiveJob
       end
     end
 
+    def concurrency_limited?
+      concurrency_key.present?
+    end
+
     private
       def concurrency_group
         compute_concurrency_parameter(self.class.concurrency_group)

data/lib/active_job/queue_adapters/solid_queue_adapter.rb

@@ -9,15 +9,15 @@ module ActiveJob
     #   Rails.application.config.active_job.queue_adapter = :solid_queue
     class SolidQueueAdapter
       def enqueue(active_job) # :nodoc:
-        SolidQueue::Job.enqueue_active_job(active_job).tap do |job|
-          active_job.provider_job_id = job.id
-        end
+        SolidQueue::Job.enqueue(active_job)
       end
 
       def enqueue_at(active_job, timestamp) # :nodoc:
-        SolidQueue::Job.enqueue_active_job(active_job, scheduled_at: Time.at(timestamp)).tap do |job|
-          active_job.provider_job_id = job.id
-        end
+        SolidQueue::Job.enqueue(active_job, scheduled_at: Time.at(timestamp))
+      end
+
+      def enqueue_all(active_jobs) # :nodoc:
+        SolidQueue::Job.enqueue_all(active_jobs)
       end
     end
   end

data/lib/generators/solid_queue/install/install_generator.rb

@@ -1,14 +1,16 @@
 # frozen_string_literal: true
 
 class SolidQueue::InstallGenerator < Rails::Generators::Base
+  source_root File.expand_path("templates", __dir__)
+
   class_option :skip_migrations, type: :boolean, default: nil, desc: "Skip migrations"
 
   def add_solid_queue
-    %w[ development test production ].each do |env_name|
-      if (env_config = Pathname(destination_root).join("config/environments/#{env_name}.rb")).exist?
-        gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
-      end
+    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
+      gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
     end
+
+    copy_file "config.yml", "config/solid_queue.yml"
   end
 
   def create_migrations

data/lib/generators/solid_queue/install/templates/config.yml

@@ -0,0 +1,18 @@
+#default: &default
+#   dispatchers:
+#     - polling_interval: 1
+#       batch_size: 500
+#   workers:
+#     - queues: "*"
+#       threads: 5
+#       processes: 1
+#       polling_interval: 0.1
+#
+# development:
+#  <<: *default
+#
+# test:
+#  <<: *default
+#
+# production:
+#  <<: *default

data/lib/puma/plugin/solid_queue.rb

@@ -6,16 +6,19 @@ Puma::Plugin.create do
   def start(launcher)
     @log_writer = launcher.log_writer
     @puma_pid = $$
-    @solid_queue_pid = fork do
-      Thread.new { monitor_puma }
-      SolidQueue::Supervisor.start(mode: :all)
-    end
 
-    launcher.events.on_stopped { stop_solid_queue }
+    launcher.events.on_booted do
+      @solid_queue_pid = fork do
+        Thread.new { monitor_puma }
+        SolidQueue::Supervisor.start(mode: :all)
+      end
 
-    in_background do
-      monitor_solid_queue
+      in_background do
+        monitor_solid_queue
+      end
     end
+
+    launcher.events.on_stopped { stop_solid_queue }
   end
 
   private

data/lib/solid_queue/configuration.rb

@@ -73,29 +73,37 @@ module SolidQueue
           .map { |options| options.dup.symbolize_keys }
       end
 
+
       def load_config_from(file_or_hash)
         case file_or_hash
-        when Pathname then load_config_file file_or_hash
-        when String   then load_config_file Pathname.new(file_or_hash)
-        when NilClass then load_config_file default_config_file
-        when Hash     then file_or_hash.dup
-        else          raise "Solid Queue cannot be initialized with #{file_or_hash.inspect}"
+        when Hash
+          file_or_hash.dup
+        when Pathname, String
+          load_config_from_file Pathname.new(file_or_hash)
+        when NilClass
+          load_config_from_env_location || load_config_from_default_location
+        else
+          raise "Solid Queue cannot be initialized with #{file_or_hash.inspect}"
         end
       end
 
-      def load_config_file(file)
-        if file.exist?
-          ActiveSupport::ConfigurationFile.parse(file).deep_symbolize_keys
-        else
-          raise "Configuration file not found in #{file}"
+      def load_config_from_env_location
+        if ENV["SOLID_QUEUE_CONFIG"].present?
+          load_config_from_file Rails.root.join(ENV["SOLID_QUEUE_CONFIG"])
         end
       end
 
-      def default_config_file
-        path_to_file = ENV["SOLID_QUEUE_CONFIG"] || DEFAULT_CONFIG_FILE_PATH
+      def load_config_from_default_location
+        Rails.root.join(DEFAULT_CONFIG_FILE_PATH).then do |config_file|
+          config_file.exist? ? load_config_from_file(config_file) : {}
+        end
+      end
 
-        Rails.root.join(path_to_file).tap do |config_file|
-          raise "Configuration for Solid Queue not found in #{config_file}" unless config_file.exist?
+      def load_config_from_file(file)
+        if file.exist?
+          ActiveSupport::ConfigurationFile.parse(file).deep_symbolize_keys
+        else
+          raise "Configuration file for Solid Queue not found in #{file}"
         end
       end
   end

data/lib/solid_queue/processes/runnable.rb

@@ -44,7 +44,9 @@ module SolidQueue::Processes
       loop do
         break if shutting_down?
 
-        run
+        wrap_in_app_executor do
+          run
+        end
       end
     ensure
       run_callbacks(:shutdown) { shutdown }

data/lib/solid_queue/supervisor.rb

@@ -79,7 +79,7 @@ module SolidQueue
       end
 
       def graceful_termination
-        procline "terminating gracefully"
+        SolidQueue.logger.info("[SolidQueue] Terminating gracefully...")
         term_forks
 
         wait_until(SolidQueue.shutdown_timeout, -> { all_forks_terminated? }) do
@@ -90,7 +90,7 @@ module SolidQueue
       end
 
       def immediate_termination
-        procline "terminating immediately"
+        SolidQueue.logger.info("[SolidQueue] Terminating immediately...")
         quit_forks
       end
 

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,29 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-12-18 00:00:00.000000000 Z
+date: 2023-12-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 7.0.3.1
+        version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 7.0.3.1
+        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: puma
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Database-backed Active Job backend.
 email:
 - rosa@37signals.com
@@ -71,6 +85,7 @@ files:
 - app/models/solid_queue/job/clearable.rb
 - app/models/solid_queue/job/concurrency_controls.rb
 - app/models/solid_queue/job/executable.rb
+- app/models/solid_queue/job/schedulable.rb
 - app/models/solid_queue/pause.rb
 - app/models/solid_queue/process.rb
 - app/models/solid_queue/process/prunable.rb
@@ -86,6 +101,7 @@ files:
 - lib/active_job/queue_adapters/solid_queue_adapter.rb
 - lib/generators/solid_queue/install/USAGE
 - lib/generators/solid_queue/install/install_generator.rb
+- lib/generators/solid_queue/install/templates/config.yml
 - lib/puma/plugin/solid_queue.rb
 - lib/solid_queue.rb
 - lib/solid_queue/app_executor.rb
@@ -127,7 +143,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.20
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: Database-backed Active Job backend.