solid_queue 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2c5bed020eb4391f1885c95f2f4373e30d1eacd544e2cfcadf131c8c09079fd0
-  data.tar.gz: 749ac036b072622cd2cc299f1fc5b1a7353a44347beccaa8407157eed8af02ef
+  metadata.gz: 490264bacfa32881edc69e3fa349081bfa1666785db38c7df55824ab6bd538b4
+  data.tar.gz: b12570f172afad018ac4709c7203276c0ba737ff90654cae02cadc3197be0357
 SHA512:
-  metadata.gz: f39c8aadb36bc8ae5556f3cbd190e6d123ca441b14374b56c207530dd2d37d8dc4880f6e977f6bc2e047b47c886768a3c86d6fefd040a447f6d8f21e28e8a273
-  data.tar.gz: d95c4be96c4388c6558a32f44690e9d92913e47bd01a379f09ef8b3e65a9098a2e578dd68885c36374b27e389c65d67e1100ad4d1c6908e6f675ede75bce8d3a
+  metadata.gz: b5909c1a726457b029d0de62bd22bf6929d84ebcddf35a56a8a0af10a0be7108ca1eb60ed48294ef77013f03d9e3c0f9b87a34d1e54ec966742abf79f5edd41f
+  data.tar.gz: daec7288a9652399f5dd75c0c6d8a5d5def60f9772a122a8b6212a675bb2d5e6ccee831156afd59b624204c694276f5011df64a122df7d44b578e848798cb5fe

data/README.md

@@ -171,6 +171,7 @@ There are several settings that control how Solid Queue works that you can set a
 - `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to `true`.
 - `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, but this will happen automatically in the near future.
 - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
+- `enqueue_after_transaction_commit`: whether the job queuing is deferred to after the current Active Record transaction is committed. The default is `false`. [Read more](https://github.com/rails/rails/pull/51426).
 
 
 ## Concurrency controls
@@ -233,7 +234,7 @@ failed_execution.retry # This will re-enqueue the job as if it was enqueued for
 failed_execution.discard # This will delete the job from the system
 ```
 
-However, we recommend taking a look at [mission_control-jobs](https://github.com/basecamp/mission_control-jobs), a dashboard where, among other things, you can examine and retry/discard failed jobs. 
+However, we recommend taking a look at [mission_control-jobs](https://github.com/rails/mission_control-jobs), a dashboard where, among other things, you can examine and retry/discard failed jobs.
 
 ## Puma plugin
 We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add
@@ -246,9 +247,12 @@ to your `puma.rb` configuration.
 ## Jobs and transactional integrity
 :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.
 
+By default, Solid Queue runs in the same DB as your app, and job enqueuing is _not_ deferred until any ongoing transaction is committed, which means that by default, you'll be taking advantage of this transactional integrity.
+
 If you prefer not to rely on this, or avoid relying on it unintentionally, you should make sure that:
-- Your jobs relying on specific records are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
-- Or, to opt out completely from this behaviour, configure a database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
+- You set [`config.active_job.enqueue_after_transaction_commit`](https://edgeguides.rubyonrails.org/configuring.html#config-active-job-enqueue-after-transaction-commit) to `always`, if you're using Rails 7.2+.
+- Or, your jobs relying on specific records are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
+- Or, you configure a database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
 
   ```ruby
   class ApplicationRecord < ActiveRecord::Base
@@ -261,13 +265,6 @@ If you prefer not to rely on this, or avoid relying on it unintentionally, you s
   config.solid_queue.connects_to = { database: { writing: :primary, reading: :replica } }
   ```
 
-## Inspiration
-
-Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.
-
-## License
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
-
 ## Recurring tasks
 Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by dispatcher processes and as such, they can be defined in the dispatcher's configuration like this:
 ```yml
@@ -312,3 +309,10 @@ You can still configure this in Solid Queue:
           schedule: "*/5 * * * *"
 ```
 and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.
+
+## Inspiration
+
+Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.
+
+## License
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/app/models/solid_queue/blocked_execution.rb

@@ -10,21 +10,25 @@ module SolidQueue
     scope :expired, -> { where(expires_at: ...Time.current) }
 
     class << self
-      def unblock(count)
-        expired.distinct.limit(count).pluck(:concurrency_key).then do |concurrency_keys|
-          release_many releasable(concurrency_keys)
+      def unblock(limit)
+        SolidQueue.instrument(:release_many_blocked, limit: limit) do |payload|
+          expired.distinct.limit(limit).pluck(:concurrency_key).then do |concurrency_keys|
+            payload[:size] = release_many releasable(concurrency_keys)
+          end
         end
       end
 
       def release_many(concurrency_keys)
         # We want to release exactly one blocked execution for each concurrency key, and we need to do it
         # one by one, locking each record and acquiring the semaphore individually for each of them:
-        Array(concurrency_keys).each { |concurrency_key| release_one(concurrency_key) }
+        Array(concurrency_keys).count { |concurrency_key| release_one(concurrency_key) }
       end
 
       def release_one(concurrency_key)
         transaction do
-          ordered.where(concurrency_key: concurrency_key).limit(1).non_blocking_lock.each(&:release)
+          if execution = ordered.where(concurrency_key: concurrency_key).limit(1).non_blocking_lock.first
+            execution.release
+          end
         end
       end
 
@@ -38,12 +42,14 @@ module SolidQueue
     end
 
     def release
-      transaction do
-        if acquire_concurrency_lock
-          promote_to_ready
-          destroy!
+      SolidQueue.instrument(:release_blocked, job_id: job.id, concurrency_key: concurrency_key, released: false) do |payload|
+        transaction do
+          if acquire_concurrency_lock
+            promote_to_ready
+            destroy!
 
-          SolidQueue.logger.debug("[SolidQueue] Unblocked job #{job.id} under #{concurrency_key}")
+            payload[:released] = true
+          end
         end
       end
     end

data/app/models/solid_queue/claimed_execution.rb

@@ -20,7 +20,12 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     end
 
     def release_all
-      includes(:job).each(&:release)
+      SolidQueue.instrument(:release_many_claimed) do |payload|
+        includes(:job).tap do |executions|
+          payload[:size] = executions.size
+          executions.each(&:release)
+        end
+      end
     end
 
     def discard_all_in_batches(*)
@@ -45,9 +50,11 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
   end
 
   def release
-    transaction do
-      job.dispatch_bypassing_concurrency_limits
-      destroy!
+    SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do
+      transaction do
+        job.dispatch_bypassing_concurrency_limits
+        destroy!
+      end
     end
   end
 

data/app/models/solid_queue/execution/dispatching.rb

@@ -9,9 +9,8 @@ module SolidQueue
         def dispatch_jobs(job_ids)
           jobs = Job.where(id: job_ids)
 
-          Job.dispatch_all(jobs).map(&:id).tap do |dispatched_job_ids|
-            where(job_id: dispatched_job_ids).order(:job_id).delete_all
-            SolidQueue.logger.info("[SolidQueue] Dispatched #{dispatched_job_ids.size} jobs")
+          Job.dispatch_all(jobs).map(&:id).then do |dispatched_job_ids|
+            where(id: where(job_id: dispatched_job_ids).pluck(:id)).delete_all
           end
         end
       end

data/app/models/solid_queue/execution.rb

@@ -13,6 +13,10 @@ module SolidQueue
     belongs_to :job
 
     class << self
+      def type
+        model_name.element.sub("_execution", "").to_sym
+      end
+
       def create_all_from_jobs(jobs)
         insert_all execution_data_from_jobs(jobs)
       end
@@ -27,25 +31,32 @@ module SolidQueue
         pending = count
         discarded = 0
 
-        loop do
-          transaction do
-            job_ids = limit(batch_size).order(:job_id).lock.pluck(:job_id)
+        SolidQueue.instrument(:discard_all, batch_size: batch_size, status: type, batches: 0, size: 0) do |payload|
+          loop do
+            transaction do
+              job_ids = limit(batch_size).order(:job_id).lock.pluck(:job_id)
+              discarded = discard_jobs job_ids
 
-            discard_jobs job_ids
-            discarded = where(job_id: job_ids).delete_all
-            pending -= discarded
-          end
+              where(job_id: job_ids).delete_all
+              pending -= discarded
+
+              payload[:size] += discarded
+              payload[:batches] += 1
+            end
 
-          break if pending <= 0 || discarded == 0
+            break if pending <= 0 || discarded == 0
+          end
         end
       end
 
       def discard_all_from_jobs(jobs)
-        transaction do
-          job_ids = lock_all_from_jobs(jobs)
+        SolidQueue.instrument(:discard_all, jobs_size: jobs.size, status: type) do |payload|
+          transaction do
+            job_ids = lock_all_from_jobs(jobs)
 
-          discard_jobs job_ids
-          where(job_id: job_ids).delete_all
+            payload[:size] = discard_jobs job_ids
+            where(job_id: job_ids).delete_all
+          end
         end
       end
 
@@ -59,10 +70,16 @@ module SolidQueue
         end
     end
 
+    def type
+      self.class.type
+    end
+
     def discard
-      with_lock do
-        job.destroy
-        destroy
+      SolidQueue.instrument(:discard, job_id: job_id, status: type) do
+        with_lock do
+          job.destroy
+          destroy
+        end
       end
     end
   end

data/app/models/solid_queue/failed_execution.rb

@@ -11,15 +11,19 @@ module SolidQueue
     attr_accessor :exception
 
     def self.retry_all(jobs)
-      transaction do
-        dispatch_jobs lock_all_from_jobs(jobs)
+      SolidQueue.instrument(:retry_all, jobs_size: jobs.size) do |payload|
+        transaction do
+          payload[:size] = dispatch_jobs lock_all_from_jobs(jobs)
+        end
       end
     end
 
     def retry
-      with_lock do
-        job.prepare_for_execution
-        destroy!
+      SolidQueue.instrument(:retry, job_id: job.id) do
+        with_lock do
+          job.prepare_for_execution
+          destroy!
+        end
       end
     end
 
@@ -33,5 +37,5 @@ module SolidQueue
           self.error = { exception_class: exception.class.name, message: exception.message, backtrace: exception.backtrace }
         end
       end
-    end
+  end
 end

data/app/models/solid_queue/job/executable.rb

@@ -93,7 +93,7 @@ module SolidQueue
         if finished?
           :finished
         elsif execution.present?
-          execution.model_name.element.sub("_execution", "").to_sym
+          execution.type
         end
       end
 

data/app/models/solid_queue/job/schedulable.rb

@@ -8,7 +8,7 @@ module SolidQueue
       included do
         has_one :scheduled_execution
 
-        scope :scheduled, -> { where.not(finished_at: nil) }
+        scope :scheduled, -> { where(finished_at: nil) }
       end
 
       class_methods do

data/app/models/solid_queue/process/prunable.rb

@@ -4,15 +4,16 @@ module SolidQueue::Process::Prunable
   extend ActiveSupport::Concern
 
   included do
-    scope :prunable, -> { where("last_heartbeat_at <= ?", SolidQueue.process_alive_threshold.ago) }
+    scope :prunable, -> { where(last_heartbeat_at: ..SolidQueue.process_alive_threshold.ago) }
   end
 
   class_methods do
     def prune
-      prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
-        batch.each do |process|
-          SolidQueue.logger.info("[SolidQueue] Pruning dead process #{process.id} - #{process.metadata}")
-          process.deregister
+      SolidQueue.instrument :prune_processes, size: 0 do |payload|
+        prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
+          payload[:size] += batch.size
+
+          batch.each { |process| process.deregister(pruned: true) }
         end
       end
     end

data/app/models/solid_queue/process.rb

@@ -12,17 +12,24 @@ class SolidQueue::Process < SolidQueue::Record
   after_destroy -> { claimed_executions.release_all }
 
   def self.register(**attributes)
-    create!(attributes.merge(last_heartbeat_at: Time.current))
+    SolidQueue.instrument :register_process, **attributes do
+      create!(attributes.merge(last_heartbeat_at: Time.current))
+    end
+  rescue Exception => error
+    SolidQueue.instrument :register_process, **attributes.merge(error: error)
+    raise
   end
 
   def heartbeat
     touch(:last_heartbeat_at)
   end
 
-  def deregister
-    destroy!
-  rescue Exception
-    SolidQueue.logger.error("[SolidQueue] Error deregistering process #{id} - #{metadata}")
-    raise
+  def deregister(pruned: false)
+    SolidQueue.instrument :deregister_process, process: self, pruned: pruned, claimed_size: claimed_executions.size do |payload|
+      destroy!
+    rescue Exception => error
+      payload[:error] = error
+      raise
+    end
   end
 end

data/app/models/solid_queue/recurring_execution.rb

@@ -7,12 +7,12 @@ module SolidQueue
     class << self
       def record(task_key, run_at, &block)
         transaction do
-          if job_id = block.call
-            create!(job_id: job_id, task_key: task_key, run_at: run_at)
+          block.call.tap do |active_job|
+            create!(job_id: active_job.provider_job_id, task_key: task_key, run_at: run_at)
           end
         end
       rescue ActiveRecord::RecordNotUnique
-        SolidQueue.logger.info("[SolidQueue] Skipped recurring task #{task_key} at #{run_at} — already dispatched")
+        # Task already dispatched
       end
 
       def clear_in_batches(batch_size: 500)

data/app/models/solid_queue/scheduled_execution.rb

@@ -16,7 +16,9 @@ module SolidQueue
           job_ids = next_batch(batch_size).non_blocking_lock.pluck(:job_id)
           if job_ids.empty? then []
           else
-            dispatch_jobs(job_ids)
+            SolidQueue.instrument(:dispatch_scheduled, batch_size: batch_size) do |payload|
+              payload[:size] = dispatch_jobs(job_ids)
+            end
           end
         end
       end

data/app/models/solid_queue/semaphore.rb

@@ -70,7 +70,7 @@ module SolidQueue
         end
 
         def limit
-          job.concurrency_limit
+          job.concurrency_limit || 1
         end
     end
   end

data/lib/active_job/queue_adapters/solid_queue_adapter.rb

@@ -8,6 +8,10 @@ module ActiveJob
     #
     #   Rails.application.config.active_job.queue_adapter = :solid_queue
     class SolidQueueAdapter
+      def enqueue_after_transaction_commit?
+        SolidQueue.enqueue_after_transaction_commit
+      end
+
       def enqueue(active_job) # :nodoc:
         SolidQueue::Job.enqueue(active_job)
       end

data/lib/generators/solid_queue/install/templates/config.yml

@@ -4,7 +4,7 @@
 #       batch_size: 500
 #   workers:
 #     - queues: "*"
-#       threads: 5
+#       threads: 3
 #       processes: 1
 #       polling_interval: 0.1
 #

data/lib/puma/plugin/solid_queue.rb

@@ -19,6 +19,7 @@ Puma::Plugin.create do
     end
 
     launcher.events.on_stopped { stop_solid_queue }
+    launcher.events.on_restart { stop_solid_queue }
   end
 
   private

data/lib/solid_queue/app_executor.rb

@@ -11,7 +11,7 @@ module SolidQueue
     end
 
     def handle_thread_error(error)
-      SolidQueue.logger.error("[SolidQueue] #{error}")
+      SolidQueue.instrument(:thread_error, error: error)
 
       if SolidQueue.on_thread_error
         SolidQueue.on_thread_error.call(error)

data/lib/solid_queue/dispatcher/recurring_task.rb

@@ -30,10 +30,16 @@ module SolidQueue
     end
 
     def enqueue(at:)
-      if using_solid_queue_adapter?
-        perform_later_and_record(run_at: at)
-      else
-        perform_later
+      SolidQueue.instrument(:enqueue_recurring_task, task: key, at: at) do |payload|
+        if using_solid_queue_adapter?
+          perform_later_and_record(run_at: at)
+        else
+          payload[:other_adapter] = true
+
+          perform_later
+        end.tap do |active_job|
+          payload[:active_job_id] = active_job&.job_id
+        end
       end
     end
 
@@ -42,7 +48,7 @@ module SolidQueue
     end
 
     def to_s
-      "#{class_name}.perform_later(#{arguments.map(&:inspect).join(",")}) [ #{parsed_schedule.original.to_s} ]"
+      "#{class_name}.perform_later(#{arguments.map(&:inspect).join(",")}) [ #{parsed_schedule.original} ]"
     end
 
     def to_h
@@ -59,7 +65,7 @@ module SolidQueue
       end
 
       def perform_later_and_record(run_at:)
-        RecurringExecution.record(key, run_at) { perform_later.provider_job_id }
+        RecurringExecution.record(key, run_at) { perform_later }
       end
 
       def perform_later
@@ -82,4 +88,4 @@ module SolidQueue
         @job_class ||= class_name.safe_constantize
       end
   end
-end
+end

data/lib/solid_queue/dispatcher.rb

@@ -19,6 +19,10 @@ module SolidQueue
       @recurring_schedule = RecurringSchedule.new(options[:recurring_tasks])
     end
 
+    def metadata
+      super.merge(batch_size: batch_size, concurrency_maintenance_interval: concurrency_maintenance&.interval, recurring_schedule: recurring_schedule.tasks.presence)
+    end
+
     private
       def poll
         batch = dispatch_next_batch
@@ -50,9 +54,5 @@ module SolidQueue
       def set_procline
         procline "waiting"
       end
-
-      def metadata
-        super.merge(batch_size: batch_size, concurrency_maintenance_interval: concurrency_maintenance&.interval, recurring_schedule: recurring_schedule.tasks.presence )
-      end
   end
 end

data/lib/solid_queue/engine.rb

@@ -18,7 +18,7 @@ module SolidQueue
 
     initializer "solid_queue.app_executor", before: :run_prepare_callbacks do |app|
       config.solid_queue.app_executor    ||= app.executor
-      config.solid_queue.on_thread_error ||= -> (exception) { Rails.error.report(exception, handled: false) }
+      config.solid_queue.on_thread_error ||= ->(exception) { Rails.error.report(exception, handled: false) }
 
       SolidQueue.app_executor = config.solid_queue.app_executor
       SolidQueue.on_thread_error = config.solid_queue.on_thread_error
@@ -26,8 +26,10 @@ module SolidQueue
 
     initializer "solid_queue.logger" do |app|
       ActiveSupport.on_load(:solid_queue) do
-        self.logger = app.logger
+        self.logger ||= app.logger
       end
+
+      SolidQueue::LogSubscriber.attach_to :solid_queue
     end
 
     initializer "solid_queue.active_job.extensions" do