solid_queue 0.1.2 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2e1ff04f60a3eee3be19e692691ef7a70cd1cfc1a2edd5f019f6bbab9b8614e6
-  data.tar.gz: bc96a464a966379d434b4e8c45c4ad24c30bf3aeed6d3b6acc60d94e1386a14e
+  metadata.gz: 0eb31439e2768af5f5d3fb9289ac51056570c762ad140bc54aba81ee770d5a12
+  data.tar.gz: 4d3e4c2608b0a3ed2de82c7712c8a524e8745bc4e2ddf6907de8b3723560ab04
 SHA512:
-  metadata.gz: 27d7bf13f3342cfccd54aa3cc3db18284d68b7fc56ae62b2c9ca66710aadb0eb8e37c48b8b268169fa37aac523cbe1cdb2dd7146fd160499d82e4521cb35d2ba
-  data.tar.gz: df8f520b23ab98bdcea47ffab06df3bfa1e946072070d251ecd0d5e44a6f7e723fdd6278344b4dbf88a8d184ce661f97440ba188267d004df83a5df66945bedc
+  metadata.gz: 0f4271aaf7b55b86d81f97bf19bbfa2b76de1ff994a6b71bb097b89ab9585212f28c0fdbd4fbb86b74c18b49cf229e7913eaa724784b3a62f000d60c2ade3263
+  data.tar.gz: a84d91b13db4a3ec96d1afa5fb5f39abc9ea0bf4bc216c14e20f4bf7e1bf2ff414e08f78166377950351b8170cbd734aa69322feb9076a67a3f9939ebacf6573

data/MIT-LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2023 37signals
+Copyright (c) 37signals, LLC
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -2,29 +2,11 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, and priorities by queue order. _Proper support for `perform_all_later`, improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon._
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon._
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
-## Usage
-To set Solid Queue as your Active Job's queue backend, you should add this to your environment config:
-```ruby
-# config/environments/production.rb
-config.active_job.queue_adapter = :solid_queue
-```
-
-Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
-
-```ruby
-# app/jobs/my_job.rb
-
-class MyJob < ApplicationJob
-  self.queue_adapter = :solid_queue
-  # ...
-end
-```
-
-## Installation
+## Installation and usage
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -41,23 +23,43 @@ Or install it yourself as:
 $ gem install solid_queue
 ```
 
-Install Migrations and Set Up Active Job Adapter
-Now, you need to install the necessary migrations and configure the Active Job's adapter. Run the following commands:
+Now, you need to install the necessary migrations and configure the Active Job's adapter. You can do both at once using the provided generator:
+
 ```bash
 $ bin/rails generate solid_queue:install
 ```
 
-or add the only the migration to your app and run it:
+This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
+
+Alternatively, you can add only the migration to your app:
 ```bash
 $ bin/rails solid_queue:install:migrations
 ```
 
-Run the Migrations (required after either of the above steps):
+And set Solid Queue as your Active Job's queue backend manually, in your environment config:
+```ruby
+# config/environments/production.rb
+config.active_job.queue_adapter = :solid_queue
+```
+
+Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+
+```ruby
+# app/jobs/my_job.rb
+
+class MyJob < ApplicationJob
+  self.queue_adapter = :solid_queue
+  # ...
+end
+```
+
+Finally, you need to run the migrations:
+
 ```bash
 $ bin/rails db:migrate
 ```
 
-With this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
 ```bash
 $ bundle exec rake solid_queue:start
 ```
@@ -65,7 +67,7 @@ $ bundle exec rake solid_queue:start
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
 ## Requirements
-Besides Rails 7, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
+Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
 
 ## Configuration
 
@@ -83,6 +85,7 @@ production:
   dispatchers:
     - polling_interval: 1
       batch_size: 500
+      concurrency_maintenance_interval: 300
   workers:
     - queues: "*"
       threads: 3
@@ -97,6 +100,7 @@ Everything is optional. If no configuration is provided, Solid Queue will run wi
 
 - `polling_interval`: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to `1` second for dispatchers and `0.1` seconds for workers.
 - `batch_size`: the dispatcher will dispatch jobs in batches of this size. The default is 500.
+- `concurrency_maintenance_interval`: the time interval in seconds that the dispatcher will wait before checking for blocked jobs that can be unblocked. Read more about [concurrency controls](#concurrency-controls) to learn more about this setting. It defaults to `600` seconds.
 - `queues`: the list of queues that workers will pick jobs from. You can use `*` to indicate all queues (which is also the default and the behaviour you'll get if you omit this). You can provide a single queue, or a list of queues as an array. Jobs will be polled from those queues in order, so for example, with `[ real_time, background ]`, no jobs will be taken from `background` unless there aren't any more jobs waiting in `real_time`. You can also provide a prefix with a wildcard to match queues starting with a prefix. For example:
 
   ```yml

data/app/models/solid_queue/blocked_execution.rb

@@ -2,7 +2,7 @@
 
 module SolidQueue
   class BlockedExecution < Execution
-    assume_attributes_from_job :concurrency_key
+    assumes_attributes_from_job :concurrency_key
     before_create :set_expires_at
 
     has_one :semaphore, foreign_key: :key, primary_key: :concurrency_key

data/app/models/solid_queue/claimed_execution.rb

@@ -23,6 +23,14 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     def release_all
       includes(:job).each(&:release)
     end
+
+    def discard_all_in_batches(*)
+      raise UndiscardableError, "Can't discard jobs in progress"
+    end
+
+    def discard_all_from_jobs(*)
+      raise UndiscardableError, "Can't discard jobs in progress"
+    end
   end
 
   def perform
@@ -39,11 +47,15 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
 
   def release
     transaction do
-      job.prepare_for_execution
+      job.dispatch_bypassing_concurrency_limits
       destroy!
     end
   end
 
+  def discard
+    raise UndiscardableError, "Can't discard a job in progress"
+  end
+
   private
     def execute
       ActiveJob::Base.execute(job.arguments)

data/app/models/solid_queue/execution/dispatching.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Execution
+    module Dispatching
+      extend ActiveSupport::Concern
+
+      class_methods do
+        def dispatch_jobs(job_ids)
+          jobs = Job.where(id: job_ids)
+
+          Job.dispatch_all(jobs).map(&:id).tap do |dispatched_job_ids|
+            where(job_id: dispatched_job_ids).delete_all
+            SolidQueue.logger.info("[SolidQueue] Dispatched #{dispatched_job_ids.size} jobs")
+          end
+        end
+      end
+    end
+  end
+end

data/app/models/solid_queue/execution/job_attributes.rb

@@ -5,17 +5,24 @@ module SolidQueue
     module JobAttributes
       extend ActiveSupport::Concern
 
-      ASSUMIBLE_ATTRIBUTES_FROM_JOB = %i[ queue_name priority ]
+      included do
+        class_attribute :assumable_attributes_from_job, instance_accessor: false, default: %i[ queue_name priority ]
+      end
 
       class_methods do
-        def assume_attributes_from_job(*attributes)
-          before_create -> { assume_attributes_from_job(ASSUMIBLE_ATTRIBUTES_FROM_JOB | attributes) }
+        def assumes_attributes_from_job(*attribute_names)
+          self.assumable_attributes_from_job |= attribute_names
+          before_create -> { assume_attributes_from_job }
+        end
+
+        def attributes_from_job(job)
+          job.attributes.symbolize_keys.slice(*assumable_attributes_from_job)
         end
       end
 
       private
-        def assume_attributes_from_job(attributes)
-          attributes.each do |attribute|
+        def assume_attributes_from_job
+          self.class.assumable_attributes_from_job.each do |attribute|
             send("#{attribute}=", job.send(attribute))
           end
         end

data/app/models/solid_queue/execution.rb

@@ -2,6 +2,8 @@
 
 module SolidQueue
   class Execution < Record
+    class UndiscardableError < StandardError; end
+
     include JobAttributes
 
     self.abstract_class = true
@@ -10,6 +12,58 @@ module SolidQueue
 
     belongs_to :job
 
-    alias_method :discard, :destroy
+    class << self
+      def create_all_from_jobs(jobs)
+        insert_all execution_data_from_jobs(jobs)
+      end
+
+      def execution_data_from_jobs(jobs)
+        jobs.collect do |job|
+          attributes_from_job(job).merge(job_id: job.id)
+        end
+      end
+
+      def discard_all_in_batches(batch_size: 500)
+        pending = count
+        discarded = 0
+
+        loop do
+          transaction do
+            job_ids = limit(batch_size).order(:job_id).lock.pluck(:job_id)
+
+            discard_jobs job_ids
+            discarded = where(job_id: job_ids).delete_all
+            pending -= discarded
+          end
+
+          break if pending <= 0 || discarded == 0
+        end
+      end
+
+      def discard_all_from_jobs(jobs)
+        transaction do
+          job_ids = lock_all_from_jobs(jobs)
+
+          discard_jobs job_ids
+          where(job_id: job_ids).delete_all
+        end
+      end
+
+      private
+        def lock_all_from_jobs(jobs)
+          where(job_id: jobs.map(&:id)).order(:job_id).lock.pluck(:job_id)
+        end
+
+        def discard_jobs(job_ids)
+          Job.where(id: job_ids).delete_all
+        end
+    end
+
+    def discard
+      with_lock do
+        job.destroy
+        destroy
+      end
+    end
   end
 end

data/app/models/solid_queue/failed_execution.rb

@@ -1,31 +1,37 @@
 # frozen_string_literal: true
 
-class SolidQueue::FailedExecution < SolidQueue::Execution
-  if Gem::Version.new(Rails.version) >= Gem::Version.new("7.1")
+module SolidQueue
+  class FailedExecution < Execution
+    include Dispatching
+
     serialize :error, coder: JSON
-  else
-    serialize :error, JSON
-  end
 
-  before_create :expand_error_details_from_exception
+    before_create :expand_error_details_from_exception
+
+    attr_accessor :exception
 
-  attr_accessor :exception
+    def self.retry_all(jobs)
+      transaction do
+        dispatch_jobs lock_all_from_jobs(jobs)
+      end
+    end
 
-  def retry
-    transaction do
-      job.prepare_for_execution
-      destroy!
+    def retry
+      with_lock do
+        job.prepare_for_execution
+        destroy!
+      end
     end
-  end
 
-  %i[ exception_class message backtrace ].each do |attribute|
-    define_method(attribute) { error.with_indifferent_access[attribute] }
-  end
+    %i[ exception_class message backtrace ].each do |attribute|
+      define_method(attribute) { error.with_indifferent_access[attribute] }
+    end
 
-  private
-    def expand_error_details_from_exception
-      if exception
-        self.error = { exception_class: exception.class.name, message: exception.message, backtrace: exception.backtrace }
+    private
+      def expand_error_details_from_exception
+        if exception
+          self.error = { exception_class: exception.class.name, message: exception.message, backtrace: exception.backtrace }
+        end
       end
     end
 end

data/app/models/solid_queue/job/clearable.rb

@@ -11,7 +11,10 @@ module SolidQueue
 
       class_methods do
         def clear_finished_in_batches(batch_size: 500, finished_before: SolidQueue.clear_finished_jobs_after.ago)
-          clearable(finished_before: finished_before).in_batches(of: batch_size).delete_all
+          loop do
+            records_deleted = clearable(finished_before: finished_before).limit(batch_size).delete_all
+            break if records_deleted == 0
+          end
         end
       end
     end

data/app/models/solid_queue/job/concurrency_controls.rb

@@ -6,9 +6,17 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        has_one :blocked_execution, dependent: :destroy
+        has_one :blocked_execution
 
         delegate :concurrency_limit, :concurrency_duration, to: :job_class
+
+        before_destroy :unblock_next_blocked_job, if: -> { concurrency_limited? && ready? }
+      end
+
+      class_methods do
+        def release_all_concurrency_locks(jobs)
+          Semaphore.signal_all(jobs.select(&:concurrency_limited?))
+        end
       end
 
       def unblock_next_blocked_job
@@ -21,6 +29,10 @@ module SolidQueue
         concurrency_key.present?
       end
 
+      def blocked?
+        blocked_execution.present?
+      end
+
       private
         def acquire_concurrency_lock
           return true unless concurrency_limited?
@@ -45,6 +57,10 @@ module SolidQueue
         def job_class
           @job_class ||= class_name.safe_constantize
         end
+
+        def execution
+          super || blocked_execution
+        end
     end
   end
 end

data/app/models/solid_queue/job/executable.rb

@@ -6,20 +6,56 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        include Clearable, ConcurrencyControls
+        include Clearable, ConcurrencyControls, Schedulable
 
-        has_one :ready_execution, dependent: :destroy
-        has_one :claimed_execution, dependent: :destroy
-        has_one :failed_execution, dependent: :destroy
-
-        has_one :scheduled_execution, dependent: :destroy
+        has_one :ready_execution
+        has_one :claimed_execution
+        has_one :failed_execution
 
         after_create :prepare_for_execution
 
         scope :finished, -> { where.not(finished_at: nil) }
+        scope :failed, -> { includes(:failed_execution).where.not(failed_execution: { id: nil }) }
       end
 
-      %w[ ready claimed failed scheduled ].each do |status|
+      class_methods do
+        def prepare_all_for_execution(jobs)
+          due, not_yet_due = jobs.partition(&:due?)
+          dispatch_all(due) + schedule_all(not_yet_due)
+        end
+
+        def dispatch_all(jobs)
+          with_concurrency_limits, without_concurrency_limits = jobs.partition(&:concurrency_limited?)
+
+          dispatch_all_at_once(without_concurrency_limits)
+          dispatch_all_one_by_one(with_concurrency_limits)
+
+          successfully_dispatched(jobs)
+        end
+
+        private
+          def dispatch_all_at_once(jobs)
+            ReadyExecution.create_all_from_jobs jobs
+          end
+
+          def dispatch_all_one_by_one(jobs)
+            jobs.each(&:dispatch)
+          end
+
+          def successfully_dispatched(jobs)
+            dispatched_and_ready(jobs) + dispatched_and_blocked(jobs)
+          end
+
+          def dispatched_and_ready(jobs)
+            where(id: ReadyExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+
+          def dispatched_and_blocked(jobs)
+            where(id: BlockedExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+      end
+
+      %w[ ready claimed failed ].each do |status|
         define_method("#{status}?") { public_send("#{status}_execution").present? }
       end
 
@@ -37,6 +73,10 @@ module SolidQueue
         end
       end
 
+      def dispatch_bypassing_concurrency_limits
+        ready
+      end
+
       def finished!
         if preserve_finished_jobs?
           touch(:finished_at)
@@ -49,35 +89,34 @@ module SolidQueue
         finished_at.present?
       end
 
-      def failed_with(exception)
-        FailedExecution.create_or_find_by!(job_id: id, exception: exception)
-      end
-
-      def discard
-        destroy unless claimed?
+      def status
+        if finished?
+          :finished
+        elsif execution.present?
+          execution.model_name.element.sub("_execution", "").to_sym
+        end
       end
 
       def retry
         failed_execution&.retry
       end
 
+      def discard
+        execution&.discard
+      end
+
       def failed_with(exception)
         FailedExecution.create_or_find_by!(job_id: id, exception: exception)
       end
 
       private
-        def due?
-          scheduled_at.nil? || scheduled_at <= Time.current
-        end
-
-        def schedule
-          ScheduledExecution.create_or_find_by!(job_id: id)
-        end
-
         def ready
           ReadyExecution.create_or_find_by!(job_id: id)
         end
 
+        def execution
+          %w[ ready claimed failed ].reduce(nil) { |acc, status| acc || public_send("#{status}_execution") }
+        end
 
         def preserve_finished_jobs?
           SolidQueue.preserve_finished_jobs

data/app/models/solid_queue/job/schedulable.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Job
+    module Schedulable
+      extend ActiveSupport::Concern
+
+      included do
+        has_one :scheduled_execution
+
+        scope :scheduled, -> { where.not(finished_at: nil) }
+      end
+
+      class_methods do
+        def schedule_all(jobs)
+          schedule_all_at_once(jobs)
+          successfully_scheduled(jobs)
+        end
+
+        private
+          def schedule_all_at_once(jobs)
+            ScheduledExecution.create_all_from_jobs(jobs)
+          end
+
+          def successfully_scheduled(jobs)
+            where(id: ScheduledExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          end
+      end
+
+      def due?
+        scheduled_at.nil? || scheduled_at <= Time.current
+      end
+
+      def scheduled?
+        scheduled_execution.present?
+      end
+
+      private
+        def schedule
+          ScheduledExecution.create_or_find_by!(job_id: id)
+        end
+
+        def execution
+          super || scheduled_execution
+        end
+    end
+  end
+end

data/app/models/solid_queue/job.rb

@@ -1,38 +1,61 @@
 # frozen_string_literal: true
 
-class SolidQueue::Job < SolidQueue::Record
-  include Executable
+module SolidQueue
+  class Job < Record
+    include Executable
 
-  if Gem::Version.new(Rails.version) >= Gem::Version.new("7.1")
     serialize :arguments, coder: JSON
-  else
-    serialize :arguments, JSON
-  end
 
-  DEFAULT_PRIORITY = 0
-  DEFAULT_QUEUE_NAME = "default"
-
-  class << self
-    def enqueue_active_job(active_job, scheduled_at: Time.current)
-      enqueue \
-        queue_name: active_job.queue_name,
-        active_job_id: active_job.job_id,
-        priority: active_job.priority,
-        scheduled_at: scheduled_at,
-        class_name: active_job.class.name,
-        arguments: active_job.serialize,
-        concurrency_key: active_job.try(:concurrency_key)
-    end
+    class << self
+      def enqueue_all(active_jobs)
+        active_jobs_by_job_id = active_jobs.index_by(&:job_id)
+
+        transaction do
+          jobs = create_all_from_active_jobs(active_jobs)
+          prepare_all_for_execution(jobs).tap do |enqueued_jobs|
+            enqueued_jobs.each do |enqueued_job|
+              active_jobs_by_job_id[enqueued_job.active_job_id].provider_job_id = enqueued_job.id
+              active_jobs_by_job_id[enqueued_job.active_job_id].successfully_enqueued = true
+            end
+          end
+        end
 
-    def enqueue(**kwargs)
-      create!(**kwargs.compact.with_defaults(defaults)).tap do
-        SolidQueue.logger.debug "[SolidQueue] Enqueued job #{kwargs}"
+        active_jobs.count(&:successfully_enqueued?)
       end
-    end
 
-    private
-      def defaults
-        { queue_name: DEFAULT_QUEUE_NAME, priority: DEFAULT_PRIORITY }
+      def enqueue(active_job, scheduled_at: Time.current)
+        active_job.scheduled_at = scheduled_at
+
+        create_from_active_job(active_job).tap do |enqueued_job|
+          active_job.provider_job_id = enqueued_job.id
+        end
       end
+
+      private
+        DEFAULT_PRIORITY = 0
+        DEFAULT_QUEUE_NAME = "default"
+
+        def create_from_active_job(active_job)
+          create!(**attributes_from_active_job(active_job))
+        end
+
+        def create_all_from_active_jobs(active_jobs)
+          job_rows = active_jobs.map { |job| attributes_from_active_job(job) }
+          insert_all(job_rows)
+          where(active_job_id: active_jobs.map(&:job_id))
+        end
+
+        def attributes_from_active_job(active_job)
+          {
+            queue_name: active_job.queue_name || DEFAULT_QUEUE_NAME,
+            active_job_id: active_job.job_id,
+            priority: active_job.priority || DEFAULT_PRIORITY,
+            scheduled_at: active_job.scheduled_at,
+            class_name: active_job.class.name,
+            arguments: active_job.serialize,
+            concurrency_key: active_job.concurrency_key
+          }
+        end
+    end
   end
 end

data/app/models/solid_queue/queue.rb

@@ -33,7 +33,7 @@ module SolidQueue
     end
 
     def clear
-      Job.where(queue_name: name).each(&:discard)
+      ReadyExecution.queued_as(name).discard_all_in_batches
     end
 
     def size

data/app/models/solid_queue/ready_execution.rb

@@ -4,7 +4,7 @@ module SolidQueue
   class ReadyExecution < Execution
     scope :queued_as, ->(queue_name) { where(queue_name: queue_name) }
 
-    assume_attributes_from_job
+    assumes_attributes_from_job
 
     class << self
       def claim(queue_list, limit, process_id)
@@ -15,6 +15,10 @@ module SolidQueue
         end
       end
 
+      def aggregated_count_across(queue_list)
+        QueueSelector.new(queue_list, self).scoped_relations.map(&:count).sum
+      end
+
       private
         def select_and_lock(queue_relation, process_id, limit)
           return [] if limit <= 0
@@ -36,6 +40,12 @@ module SolidQueue
             where(job_id: claimed.pluck(:job_id)).delete_all
           end
         end
+
+
+        def discard_jobs(job_ids)
+          Job.release_all_concurrency_locks Job.where(id: job_ids)
+          super
+        end
     end
   end
 end

data/app/models/solid_queue/scheduled_execution.rb

@@ -2,11 +2,13 @@
 
 module SolidQueue
   class ScheduledExecution < Execution
+    include Dispatching
+
     scope :due, -> { where(scheduled_at: ..Time.current) }
     scope :ordered, -> { order(scheduled_at: :asc, priority: :asc) }
     scope :next_batch, ->(batch_size) { due.ordered.limit(batch_size) }
 
-    assume_attributes_from_job :scheduled_at
+    assumes_attributes_from_job :scheduled_at
 
     class << self
       def dispatch_next_batch(batch_size)
@@ -14,52 +16,10 @@ module SolidQueue
           job_ids = next_batch(batch_size).non_blocking_lock.pluck(:job_id)
           if job_ids.empty? then []
           else
-            dispatch_batch(job_ids)
+            dispatch_jobs(job_ids)
           end
         end
       end
-
-      private
-        def dispatch_batch(job_ids)
-          jobs = Job.where(id: job_ids)
-          with_concurrency_limits, without_concurrency_limits = jobs.partition(&:concurrency_limited?)
-
-          dispatch_at_once(without_concurrency_limits)
-          dispatch_one_by_one(with_concurrency_limits)
-
-          successfully_dispatched(job_ids).tap do |dispatched_job_ids|
-            where(job_id: dispatched_job_ids).delete_all
-            SolidQueue.logger.info("[SolidQueue] Dispatched scheduled batch with #{dispatched_job_ids.size} jobs")
-          end
-        end
-
-        def dispatch_at_once(jobs)
-          ReadyExecution.insert_all ready_rows_from_batch(jobs)
-        end
-
-        def dispatch_one_by_one(jobs)
-          jobs.each(&:dispatch)
-        end
-
-        def ready_rows_from_batch(jobs)
-          prepared_at = Time.current
-
-          jobs.map do |job|
-            { job_id: job.id, queue_name: job.queue_name, priority: job.priority, created_at: prepared_at }
-          end
-        end
-
-        def successfully_dispatched(job_ids)
-          dispatched_and_ready(job_ids) + dispatched_and_blocked(job_ids)
-        end
-
-        def dispatched_and_ready(job_ids)
-          ReadyExecution.where(job_id: job_ids).pluck(:job_id)
-        end
-
-        def dispatched_and_blocked(job_ids)
-          BlockedExecution.where(job_id: job_ids).pluck(:job_id)
-        end
     end
   end
 end

data/app/models/solid_queue/semaphore.rb

@@ -1,65 +1,94 @@
 # frozen_string_literal: true
 
-class SolidQueue::Semaphore < SolidQueue::Record
-  scope :available, -> { where("value > 0") }
-  scope :expired, -> { where(expires_at: ...Time.current) }
+module SolidQueue
+  class Semaphore < Record
+    scope :available, -> { where("value > 0") }
+    scope :expired, -> { where(expires_at: ...Time.current) }
 
-  class << self
-    def wait(job)
-      Proxy.new(job, self).wait
-    end
-
-    def signal(job)
-      Proxy.new(job, self).signal
-    end
-  end
-
-  class Proxy
-    def initialize(job, proxied_class)
-      @job = job
-      @proxied_class = proxied_class
-    end
+    class << self
+      def wait(job)
+        Proxy.new(job).wait
+      end
 
-    def wait
-      if semaphore = proxied_class.find_by(key: key)
-        semaphore.value > 0 && attempt_decrement
-      else
-        attempt_creation
+      def signal(job)
+        Proxy.new(job).signal
       end
-    end
 
-    def signal
-      attempt_increment
+      def signal_all(jobs)
+        Proxy.signal_all(jobs)
+      end
     end
 
-    private
-      attr_reader :job, :proxied_class
-
-      def attempt_creation
-        proxied_class.create!(key: key, value: limit - 1, expires_at: expires_at)
-        true
-      rescue ActiveRecord::RecordNotUnique
-        attempt_decrement
+    class Proxy
+      def self.signal_all(jobs)
+        Semaphore.where(key: jobs.map(&:concurrency_key)).update_all("value = value + 1")
       end
 
-      def attempt_decrement
-        proxied_class.available.where(key: key).update_all([ "value = value - 1, expires_at = ?", expires_at ]) > 0
+      def initialize(job)
+        @job = job
+        @retries = 0
       end
 
-      def attempt_increment
-        proxied_class.where(key: key, value: ...limit).update_all([ "value = value + 1, expires_at = ?", expires_at ]) > 0
+      def wait
+        if semaphore = Semaphore.find_by(key: key)
+          semaphore.value > 0 && attempt_decrement
+        else
+          attempt_creation
+        end
       end
 
-      def key
-        job.concurrency_key
+      def signal
+        attempt_increment
       end
 
-      def expires_at
-        job.concurrency_duration.from_now
-      end
+      private
+        attr_accessor :job, :retries
 
-      def limit
-        job.concurrency_limit
-      end
+        def attempt_creation
+          Semaphore.create!(key: key, value: limit - 1, expires_at: expires_at)
+          true
+        rescue ActiveRecord::RecordNotUnique
+          attempt_decrement
+        end
+
+        def attempt_decrement
+          Semaphore.available.where(key: key).update_all([ "value = value - 1, expires_at = ?", expires_at ]) > 0
+        rescue ActiveRecord::Deadlocked
+          if retriable? then attempt_retry
+          else
+            raise
+          end
+        end
+
+        def attempt_increment
+          Semaphore.where(key: key, value: ...limit).update_all([ "value = value + 1, expires_at = ?", expires_at ]) > 0
+        end
+
+        def attempt_retry
+          self.retries += 1
+
+          if semaphore = Semaphore.find_by(key: key)
+            semaphore.value > 0 && attempt_decrement
+          end
+        end
+
+        MAX_RETRIES = 1
+
+        def retriable?
+          retries < MAX_RETRIES
+        end
+
+        def key
+          job.concurrency_key
+        end
+
+        def expires_at
+          job.concurrency_duration.from_now
+        end
+
+        def limit
+          job.concurrency_limit
+        end
+    end
   end
 end

data/db/migrate/20240110143450_add_missing_index_to_blocked_executions.rb

@@ -0,0 +1,5 @@
+class AddMissingIndexToBlockedExecutions < ActiveRecord::Migration[7.1]
+  def change
+    add_index :solid_queue_blocked_executions, [ :concurrency_key, :priority, :job_id ], name: "index_solid_queue_blocked_executions_for_release"
+  end
+end

data/lib/active_job/concurrency_controls.rb

@@ -36,6 +36,10 @@ module ActiveJob
       end
     end
 
+    def concurrency_limited?
+      concurrency_key.present?
+    end
+
     private
       def concurrency_group
         compute_concurrency_parameter(self.class.concurrency_group)

data/lib/active_job/queue_adapters/solid_queue_adapter.rb

@@ -9,15 +9,15 @@ module ActiveJob
     #   Rails.application.config.active_job.queue_adapter = :solid_queue
     class SolidQueueAdapter
       def enqueue(active_job) # :nodoc:
-        SolidQueue::Job.enqueue_active_job(active_job).tap do |job|
-          active_job.provider_job_id = job.id
-        end
+        SolidQueue::Job.enqueue(active_job)
       end
 
       def enqueue_at(active_job, timestamp) # :nodoc:
-        SolidQueue::Job.enqueue_active_job(active_job, scheduled_at: Time.at(timestamp)).tap do |job|
-          active_job.provider_job_id = job.id
-        end
+        SolidQueue::Job.enqueue(active_job, scheduled_at: Time.at(timestamp))
+      end
+
+      def enqueue_all(active_jobs) # :nodoc:
+        SolidQueue::Job.enqueue_all(active_jobs)
       end
     end
   end

data/lib/active_job/uniqueness.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module ActiveJob
+  module Uniqueness
+    extend ActiveSupport::Concern
+
+    DEFAULT_UNIQUENESS_GROUP = ->(*) { self.class.name }
+
+    included do
+      class_attribute :uniqueness_key, instance_accessor: false
+      class_attribute :uniqueness_group, default: DEFAULT_UNIQUENESS_GROUP, instance_accessor: false
+
+      class_attribute :uniqueness_duration
+    end
+
+    class_methods do
+      def enqueued_uniquely_by(key:, group: DEFAULT_UNIQUENESS_GROUP, duration: SolidQueue.default_uniqueness_period)
+        self.uniqueness_key = key
+        self.uniqueness_group = group
+        self.uniqueness_duration = duration
+      end
+    end
+
+    def uniqueness_key
+      if self.class.uniqueness_key
+        param = compute_concurrency_parameter(self.class.concurrency_key)
+
+        case param
+        when ActiveRecord::Base
+          [ concurrency_group, param.class.name, param.id ]
+        else
+          [ concurrency_group, param ]
+        end.compact.join("/")
+      end
+    end
+
+    def enqueued_uniquely?
+      uniqueness_key.present?
+    end
+  end
+end

data/lib/generators/solid_queue/install/USAGE

@@ -4,6 +4,6 @@ Description:
 Example:
     bin/rails generate solid_queue:install
 
-    This will create:
+    This will perform the following:
         Installs solid_queue migrations
-        Replaces Active Job's adapter in envionment configuration
+        Replaces Active Job's adapter in environment configuration

data/lib/generators/solid_queue/install/install_generator.rb

@@ -6,10 +6,8 @@ class SolidQueue::InstallGenerator < Rails::Generators::Base
   class_option :skip_migrations, type: :boolean, default: nil, desc: "Skip migrations"
 
   def add_solid_queue
-    %w[ development test production ].each do |env_name|
-      if (env_config = Pathname(destination_root).join("config/environments/#{env_name}.rb")).exist?
-        gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
-      end
+    if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
+      gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
     end
 
     copy_file "config.yml", "config/solid_queue.yml"

data/lib/generators/solid_queue/install/templates/config.yml

@@ -1,4 +1,4 @@
-#default: &default
+# default: &default
 #   dispatchers:
 #     - polling_interval: 1
 #       batch_size: 500

data/lib/puma/plugin/solid_queue.rb

@@ -6,16 +6,19 @@ Puma::Plugin.create do
   def start(launcher)
     @log_writer = launcher.log_writer
     @puma_pid = $$
-    @solid_queue_pid = fork do
-      Thread.new { monitor_puma }
-      SolidQueue::Supervisor.start(mode: :all)
-    end
 
-    launcher.events.on_stopped { stop_solid_queue }
+    launcher.events.on_booted do
+      @solid_queue_pid = fork do
+        Thread.new { monitor_puma }
+        SolidQueue::Supervisor.start(mode: :all)
+      end
 
-    in_background do
-      monitor_solid_queue
+      in_background do
+        monitor_solid_queue
+      end
     end
+
+    launcher.events.on_stopped { stop_solid_queue }
   end
 
   private

data/lib/solid_queue/dispatcher/scheduled_executions_dispatcher.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Dispatcher::ScheduledExecutionsDispatcher < Dispatcher
+  end
+end

data/lib/solid_queue/processes/runnable.rb

@@ -4,6 +4,8 @@ module SolidQueue::Processes
   module Runnable
     include Supervised
 
+    attr_writer :mode
+
     def start
       @stopping = false
 
@@ -19,8 +21,6 @@ module SolidQueue::Processes
     end
 
   private
-    attr_writer :mode
-
     DEFAULT_MODE = :async
 
     def mode
@@ -44,7 +44,9 @@ module SolidQueue::Processes
       loop do
         break if shutting_down?
 
-        run
+        wrap_in_app_executor do
+          run
+        end
       end
     ensure
       run_callbacks(:shutdown) { shutdown }

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.1.2"
+  VERSION = "0.2.1"
 end

data/lib/solid_queue/worker.rb

@@ -44,7 +44,7 @@ module SolidQueue
       end
 
       def all_work_completed?
-        SolidQueue::ReadyExecution.queued_as(queues).empty?
+        SolidQueue::ReadyExecution.aggregated_count_across(queues).zero?
       end
 
       def metadata

metadata

@@ -1,29 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.2.1
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-12-21 00:00:00.000000000 Z
+date: 2024-01-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 7.0.3.1
+        version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 7.0.3.1
+        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: puma
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Database-backed Active Job backend.
 email:
 - rosa@37signals.com
@@ -65,12 +79,14 @@ files:
 - app/models/solid_queue/blocked_execution.rb
 - app/models/solid_queue/claimed_execution.rb
 - app/models/solid_queue/execution.rb
+- app/models/solid_queue/execution/dispatching.rb
 - app/models/solid_queue/execution/job_attributes.rb
 - app/models/solid_queue/failed_execution.rb
 - app/models/solid_queue/job.rb
 - app/models/solid_queue/job/clearable.rb
 - app/models/solid_queue/job/concurrency_controls.rb
 - app/models/solid_queue/job/executable.rb
+- app/models/solid_queue/job/schedulable.rb
 - app/models/solid_queue/pause.rb
 - app/models/solid_queue/process.rb
 - app/models/solid_queue/process/prunable.rb
@@ -82,8 +98,10 @@ files:
 - app/models/solid_queue/semaphore.rb
 - config/routes.rb
 - db/migrate/20231211200639_create_solid_queue_tables.rb
+- db/migrate/20240110143450_add_missing_index_to_blocked_executions.rb
 - lib/active_job/concurrency_controls.rb
 - lib/active_job/queue_adapters/solid_queue_adapter.rb
+- lib/active_job/uniqueness.rb
 - lib/generators/solid_queue/install/USAGE
 - lib/generators/solid_queue/install/install_generator.rb
 - lib/generators/solid_queue/install/templates/config.yml
@@ -92,6 +110,7 @@ files:
 - lib/solid_queue/app_executor.rb
 - lib/solid_queue/configuration.rb
 - lib/solid_queue/dispatcher.rb
+- lib/solid_queue/dispatcher/scheduled_executions_dispatcher.rb
 - lib/solid_queue/engine.rb
 - lib/solid_queue/pool.rb
 - lib/solid_queue/processes/base.rb
@@ -128,7 +147,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.20
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: Database-backed Active Job backend.