solid_queue 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2e1ff04f60a3eee3be19e692691ef7a70cd1cfc1a2edd5f019f6bbab9b8614e6
-  data.tar.gz: bc96a464a966379d434b4e8c45c4ad24c30bf3aeed6d3b6acc60d94e1386a14e
+  metadata.gz: 72d9dc2e30127c7b1fe101648d4d18b04af2d33491c729816e25dcf488246b54
+  data.tar.gz: e39dcad251d4124355adcedfa9cb1ee88eea12043eb8505568332d9fcacee428
 SHA512:
-  metadata.gz: 27d7bf13f3342cfccd54aa3cc3db18284d68b7fc56ae62b2c9ca66710aadb0eb8e37c48b8b268169fa37aac523cbe1cdb2dd7146fd160499d82e4521cb35d2ba
-  data.tar.gz: df8f520b23ab98bdcea47ffab06df3bfa1e946072070d251ecd0d5e44a6f7e723fdd6278344b4dbf88a8d184ce661f97440ba188267d004df83a5df66945bedc
+  metadata.gz: 967732a5e679269543869ca64aa410bfc43c935ad3fb6856de0860bbe5cee58716e389b5935e48a91c6c658a84fa79aff9f5eae93ba3931344d31aff0d45c8fc
+  data.tar.gz: 523af9f1b26a137d99ade0805bdc955cb81584971308928d5e16e83d3c46861947deaaa95ff96775d7eceb913f3d8995eca14707cda4533f7c3ee39f20afb650

data/README.md

@@ -2,29 +2,11 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, and priorities by queue order. _Proper support for `perform_all_later`, improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon._
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon._
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
-## Usage
-To set Solid Queue as your Active Job's queue backend, you should add this to your environment config:
-```ruby
-# config/environments/production.rb
-config.active_job.queue_adapter = :solid_queue
-```
-
-Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
-
-```ruby
-# app/jobs/my_job.rb
-
-class MyJob < ApplicationJob
-  self.queue_adapter = :solid_queue
-  # ...
-end
-```
-
-## Installation
+## Installation and usage
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -41,23 +23,43 @@ Or install it yourself as:
 $ gem install solid_queue
 ```
 
-Install Migrations and Set Up Active Job Adapter
-Now, you need to install the necessary migrations and configure the Active Job's adapter. Run the following commands:
+Now, you need to install the necessary migrations and configure the Active Job's adapter. You can do both at once using the provided generator:
+
 ```bash
 $ bin/rails generate solid_queue:install
 ```
 
-or add the only the migration to your app and run it:
+This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
+
+Alternatively, you can add the only the migration to your app:
 ```bash
 $ bin/rails solid_queue:install:migrations
 ```
 
-Run the Migrations (required after either of the above steps):
+And set Solid Queue as your Active Job's queue backend manually, in your environment config:
+```ruby
+# config/environments/production.rb
+config.active_job.queue_adapter = :solid_queue
+```
+
+Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+
+```ruby
+# app/jobs/my_job.rb
+
+class MyJob < ApplicationJob
+  self.queue_adapter = :solid_queue
+  # ...
+end
+```
+
+Finally, you need to run the migrations:
+
 ```bash
 $ bin/rails db:migrate
 ```
 
-With this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
 ```bash
 $ bundle exec rake solid_queue:start
 ```
@@ -65,7 +67,7 @@ $ bundle exec rake solid_queue:start
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
 ## Requirements
-Besides Rails 7, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
+Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
 
 ## Configuration
 
@@ -83,6 +85,7 @@ production:
   dispatchers:
     - polling_interval: 1
       batch_size: 500
+      concurrency_maintenance_interval: 300
   workers:
     - queues: "*"
       threads: 3
@@ -97,6 +100,7 @@ Everything is optional. If no configuration is provided, Solid Queue will run wi
 
 - `polling_interval`: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to `1` second for dispatchers and `0.1` seconds for workers.
 - `batch_size`: the dispatcher will dispatch jobs in batches of this size. The default is 500.
+- `concurrency_maintenance_interval`: the time interval in seconds that the dispatcher will wait before checking for blocked jobs that can be unblocked. Read more about [concurrency controls](#concurrency-controls) to learn more about this setting. It defaults to `600` seconds.
 - `queues`: the list of queues that workers will pick jobs from. You can use `*` to indicate all queues (which is also the default and the behaviour you'll get if you omit this). You can provide a single queue, or a list of queues as an array. Jobs will be polled from those queues in order, so for example, with `[ real_time, background ]`, no jobs will be taken from `background` unless there aren't any more jobs waiting in `real_time`. You can also provide a prefix with a wildcard to match queues starting with a prefix. For example:
 
   ```yml

data/app/models/solid_queue/blocked_execution.rb

@@ -2,7 +2,7 @@
 
 module SolidQueue
   class BlockedExecution < Execution
-    assume_attributes_from_job :concurrency_key
+    assumes_attributes_from_job :concurrency_key
     before_create :set_expires_at
 
     has_one :semaphore, foreign_key: :key, primary_key: :concurrency_key

data/app/models/solid_queue/execution/job_attributes.rb

@@ -5,17 +5,24 @@ module SolidQueue
     module JobAttributes
       extend ActiveSupport::Concern
 
-      ASSUMIBLE_ATTRIBUTES_FROM_JOB = %i[ queue_name priority ]
+      included do
+        class_attribute :assumible_attributes_from_job, instance_accessor: false, default: %i[ queue_name priority ]
+      end
 
       class_methods do
-        def assume_attributes_from_job(*attributes)
-          before_create -> { assume_attributes_from_job(ASSUMIBLE_ATTRIBUTES_FROM_JOB | attributes) }
+        def assumes_attributes_from_job(*attribute_names)
+          self.assumible_attributes_from_job |= attribute_names
+          before_create -> { assume_attributes_from_job }
+        end
+
+        def attributes_from_job(job)
+          job.attributes.symbolize_keys.slice(*assumible_attributes_from_job)
         end
       end
 
       private
-        def assume_attributes_from_job(attributes)
-          attributes.each do |attribute|
+        def assume_attributes_from_job
+          self.class.assumible_attributes_from_job.each do |attribute|
             send("#{attribute}=", job.send(attribute))
           end
         end

data/app/models/solid_queue/execution.rb

@@ -11,5 +11,17 @@ module SolidQueue
     belongs_to :job
 
     alias_method :discard, :destroy
+
+    class << self
+      def create_all_from_jobs(jobs)
+        insert_all execution_data_from_jobs(jobs)
+      end
+
+      def execution_data_from_jobs(jobs)
+        jobs.collect do |job|
+          attributes_from_job(job).merge(job_id: job.id)
+        end
+      end
+    end
   end
 end

data/app/models/solid_queue/failed_execution.rb

@@ -1,11 +1,7 @@
 # frozen_string_literal: true
 
 class SolidQueue::FailedExecution < SolidQueue::Execution
-  if Gem::Version.new(Rails.version) >= Gem::Version.new("7.1")
-    serialize :error, coder: JSON
-  else
-    serialize :error, JSON
-  end
+  serialize :error, coder: JSON
 
   before_create :expand_error_details_from_exception
 

data/app/models/solid_queue/job/executable.rb

@@ -6,20 +6,56 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        include Clearable, ConcurrencyControls
+        include Clearable, ConcurrencyControls, Schedulable
 
         has_one :ready_execution, dependent: :destroy
         has_one :claimed_execution, dependent: :destroy
         has_one :failed_execution, dependent: :destroy
 
-        has_one :scheduled_execution, dependent: :destroy
-
         after_create :prepare_for_execution
 
         scope :finished, -> { where.not(finished_at: nil) }
+        scope :failed, -> { includes(:failed_execution).where.not(failed_execution: { id: nil }) }
       end
 
-      %w[ ready claimed failed scheduled ].each do |status|
+      class_methods do
+        def prepare_all_for_execution(jobs)
+          due, not_yet_due = jobs.partition(&:due?)
+          dispatch_all(due) + schedule_all(not_yet_due)
+        end
+
+        def dispatch_all(jobs)
+          with_concurrency_limits, without_concurrency_limits = jobs.partition(&:concurrency_limited?)
+
+          dispatch_all_at_once(without_concurrency_limits)
+          dispatch_all_one_by_one(with_concurrency_limits)
+
+          successfully_dispatched(jobs)
+        end
+
+        private
+          def dispatch_all_at_once(jobs)
+            ReadyExecution.create_all_from_jobs jobs
+          end
+
+          def dispatch_all_one_by_one(jobs)
+            jobs.each(&:dispatch)
+          end
+
+          def successfully_dispatched(jobs)
+            dispatched_and_ready(jobs) + dispatched_and_blocked(jobs)
+          end
+
+          def dispatched_and_ready(jobs)
+            where(id: ReadyExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+
+          def dispatched_and_blocked(jobs)
+            where(id: BlockedExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+      end
+
+      %w[ ready claimed failed ].each do |status|
         define_method("#{status}?") { public_send("#{status}_execution").present? }
       end
 
@@ -49,8 +85,8 @@ module SolidQueue
         finished_at.present?
       end
 
-      def failed_with(exception)
-        FailedExecution.create_or_find_by!(job_id: id, exception: exception)
+      def due?
+        scheduled_at.nil? || scheduled_at <= Time.current
       end
 
       def discard
@@ -66,14 +102,6 @@ module SolidQueue
       end
 
       private
-        def due?
-          scheduled_at.nil? || scheduled_at <= Time.current
-        end
-
-        def schedule
-          ScheduledExecution.create_or_find_by!(job_id: id)
-        end
-
         def ready
           ReadyExecution.create_or_find_by!(job_id: id)
         end

data/app/models/solid_queue/job/schedulable.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Job
+    module Schedulable
+      extend ActiveSupport::Concern
+
+      included do
+        has_one :scheduled_execution, dependent: :destroy
+
+        scope :scheduled, -> { where.not(finished_at: nil) }
+      end
+
+      class_methods do
+        def schedule_all(jobs)
+          schedule_all_at_once(jobs)
+          successfully_scheduled(jobs)
+        end
+
+        private
+          def schedule_all_at_once(jobs)
+            ScheduledExecution.create_all_from_jobs(jobs)
+          end
+
+          def successfully_scheduled(jobs)
+            where(id: ScheduledExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+      end
+
+      def due?
+        scheduled_at.nil? || scheduled_at <= Time.current
+      end
+
+      private
+        def schedule
+          ScheduledExecution.create_or_find_by!(job_id: id)
+        end
+    end
+  end
+end

data/app/models/solid_queue/job.rb

@@ -1,38 +1,58 @@
 # frozen_string_literal: true
 
-class SolidQueue::Job < SolidQueue::Record
-  include Executable
+module SolidQueue
+  class Job < Record
+    include Executable
 
-  if Gem::Version.new(Rails.version) >= Gem::Version.new("7.1")
     serialize :arguments, coder: JSON
-  else
-    serialize :arguments, JSON
-  end
 
-  DEFAULT_PRIORITY = 0
-  DEFAULT_QUEUE_NAME = "default"
-
-  class << self
-    def enqueue_active_job(active_job, scheduled_at: Time.current)
-      enqueue \
-        queue_name: active_job.queue_name,
-        active_job_id: active_job.job_id,
-        priority: active_job.priority,
-        scheduled_at: scheduled_at,
-        class_name: active_job.class.name,
-        arguments: active_job.serialize,
-        concurrency_key: active_job.try(:concurrency_key)
-    end
+    class << self
+      def enqueue_all(active_jobs)
+        active_jobs_by_job_id = active_jobs.index_by(&:job_id)
 
-    def enqueue(**kwargs)
-      create!(**kwargs.compact.with_defaults(defaults)).tap do
-        SolidQueue.logger.debug "[SolidQueue] Enqueued job #{kwargs}"
+        transaction do
+          jobs = create_all_from_active_jobs(active_jobs)
+          prepare_all_for_execution(jobs).tap do |enqueued_jobs|
+            enqueued_jobs.each do |enqueued_job|
+              active_jobs_by_job_id[enqueued_job.active_job_id].provider_job_id = enqueued_job.id
+            end
+          end
+        end
       end
-    end
 
-    private
-      def defaults
-        { queue_name: DEFAULT_QUEUE_NAME, priority: DEFAULT_PRIORITY }
+      def enqueue(active_job, scheduled_at: Time.current)
+        active_job.scheduled_at = scheduled_at
+
+        create_from_active_job(active_job).tap do |enqueued_job|
+          active_job.provider_job_id = enqueued_job.id
+        end
       end
+
+      private
+        DEFAULT_PRIORITY = 0
+        DEFAULT_QUEUE_NAME = "default"
+
+        def create_from_active_job(active_job)
+          create!(**attributes_from_active_job(active_job))
+        end
+
+        def create_all_from_active_jobs(active_jobs)
+          job_rows = active_jobs.map { |job| attributes_from_active_job(job) }
+          insert_all(job_rows)
+          where(active_job_id: active_jobs.map(&:job_id))
+        end
+
+        def attributes_from_active_job(active_job)
+          {
+            queue_name: active_job.queue_name || DEFAULT_QUEUE_NAME,
+            active_job_id: active_job.job_id,
+            priority: active_job.priority || DEFAULT_PRIORITY,
+            scheduled_at: active_job.scheduled_at,
+            class_name: active_job.class.name,
+            arguments: active_job.serialize,
+            concurrency_key: active_job.concurrency_key
+          }
+        end
+    end
   end
 end

data/app/models/solid_queue/ready_execution.rb

@@ -4,7 +4,7 @@ module SolidQueue
   class ReadyExecution < Execution
     scope :queued_as, ->(queue_name) { where(queue_name: queue_name) }
 
-    assume_attributes_from_job
+    assumes_attributes_from_job
 
     class << self
       def claim(queue_list, limit, process_id)

data/app/models/solid_queue/scheduled_execution.rb

@@ -6,7 +6,7 @@ module SolidQueue
     scope :ordered, -> { order(scheduled_at: :asc, priority: :asc) }
     scope :next_batch, ->(batch_size) { due.ordered.limit(batch_size) }
 
-    assume_attributes_from_job :scheduled_at
+    assumes_attributes_from_job :scheduled_at
 
     class << self
       def dispatch_next_batch(batch_size)
@@ -22,44 +22,12 @@ module SolidQueue
       private
         def dispatch_batch(job_ids)
           jobs = Job.where(id: job_ids)
-          with_concurrency_limits, without_concurrency_limits = jobs.partition(&:concurrency_limited?)
 
-          dispatch_at_once(without_concurrency_limits)
-          dispatch_one_by_one(with_concurrency_limits)
-
-          successfully_dispatched(job_ids).tap do |dispatched_job_ids|
+          Job.dispatch_all(jobs).map(&:id).tap do |dispatched_job_ids|
             where(job_id: dispatched_job_ids).delete_all
             SolidQueue.logger.info("[SolidQueue] Dispatched scheduled batch with #{dispatched_job_ids.size} jobs")
           end
         end
-
-        def dispatch_at_once(jobs)
-          ReadyExecution.insert_all ready_rows_from_batch(jobs)
-        end
-
-        def dispatch_one_by_one(jobs)
-          jobs.each(&:dispatch)
-        end
-
-        def ready_rows_from_batch(jobs)
-          prepared_at = Time.current
-
-          jobs.map do |job|
-            { job_id: job.id, queue_name: job.queue_name, priority: job.priority, created_at: prepared_at }
-          end
-        end
-
-        def successfully_dispatched(job_ids)
-          dispatched_and_ready(job_ids) + dispatched_and_blocked(job_ids)
-        end
-
-        def dispatched_and_ready(job_ids)
-          ReadyExecution.where(job_id: job_ids).pluck(:job_id)
-        end
-
-        def dispatched_and_blocked(job_ids)
-          BlockedExecution.where(job_id: job_ids).pluck(:job_id)
-        end
     end
   end
 end

data/lib/active_job/concurrency_controls.rb

@@ -36,6 +36,10 @@ module ActiveJob
       end
     end
 
+    def concurrency_limited?
+      concurrency_key.present?
+    end
+
     private
       def concurrency_group
         compute_concurrency_parameter(self.class.concurrency_group)

data/lib/active_job/queue_adapters/solid_queue_adapter.rb

@@ -9,15 +9,15 @@ module ActiveJob
     #   Rails.application.config.active_job.queue_adapter = :solid_queue
     class SolidQueueAdapter
       def enqueue(active_job) # :nodoc:
-        SolidQueue::Job.enqueue_active_job(active_job).tap do |job|
-          active_job.provider_job_id = job.id
-        end
+        SolidQueue::Job.enqueue(active_job)
       end
 
       def enqueue_at(active_job, timestamp) # :nodoc:
-        SolidQueue::Job.enqueue_active_job(active_job, scheduled_at: Time.at(timestamp)).tap do |job|
-          active_job.provider_job_id = job.id
-        end
+        SolidQueue::Job.enqueue(active_job, scheduled_at: Time.at(timestamp))
+      end
+
+      def enqueue_all(active_jobs) # :nodoc:
+        SolidQueue::Job.enqueue_all(active_jobs)
       end
     end
   end

data/lib/generators/solid_queue/install/install_generator.rb

@@ -6,10 +6,8 @@ class SolidQueue::InstallGenerator < Rails::Generators::Base
   class_option :skip_migrations, type: :boolean, default: nil, desc: "Skip migrations"
 
   def add_solid_queue
-    %w[ development test production ].each do |env_name|
-      if (env_config = Pathname(destination_root).join("config/environments/#{env_name}.rb")).exist?
-        gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
-      end
+    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
+      gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
     end
 
     copy_file "config.yml", "config/solid_queue.yml"

data/lib/puma/plugin/solid_queue.rb

@@ -6,16 +6,19 @@ Puma::Plugin.create do
   def start(launcher)
     @log_writer = launcher.log_writer
     @puma_pid = $$
-    @solid_queue_pid = fork do
-      Thread.new { monitor_puma }
-      SolidQueue::Supervisor.start(mode: :all)
-    end
 
-    launcher.events.on_stopped { stop_solid_queue }
+    launcher.events.on_booted do
+      @solid_queue_pid = fork do
+        Thread.new { monitor_puma }
+        SolidQueue::Supervisor.start(mode: :all)
+      end
 
-    in_background do
-      monitor_solid_queue
+      in_background do
+        monitor_solid_queue
+      end
     end
+
+    launcher.events.on_stopped { stop_solid_queue }
   end
 
   private

data/lib/solid_queue/processes/runnable.rb

@@ -44,7 +44,9 @@ module SolidQueue::Processes
       loop do
         break if shutting_down?
 
-        run
+        wrap_in_app_executor do
+          run
+        end
       end
     ensure
       run_callbacks(:shutdown) { shutdown }

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.1.2"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,29 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.2.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-12-21 00:00:00.000000000 Z
+date: 2023-12-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 7.0.3.1
+        version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 7.0.3.1
+        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: puma
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Database-backed Active Job backend.
 email:
 - rosa@37signals.com
@@ -71,6 +85,7 @@ files:
 - app/models/solid_queue/job/clearable.rb
 - app/models/solid_queue/job/concurrency_controls.rb
 - app/models/solid_queue/job/executable.rb
+- app/models/solid_queue/job/schedulable.rb
 - app/models/solid_queue/pause.rb
 - app/models/solid_queue/process.rb
 - app/models/solid_queue/process/prunable.rb
@@ -128,7 +143,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.20
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: Database-backed Active Job backend.