solid_queue 0.3.2 → 0.3.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0b3a20c6db61dce133b76763ea92b6a5b47f99b0b42d42a3e7360a616b148242
-  data.tar.gz: 5cfbd1488c274f304ab92d989703b395ceb5d7b6f277b20d053e84de0f3dd649
+  metadata.gz: b4a85fb424e127543352846ee1b0b4aa958d4c77a7a9e1c26a8082464ae7bebc
+  data.tar.gz: f15b4db5c77c14af4b989fc8abe2670fd49006853ca9bfce0080e8b15df733c3
 SHA512:
-  metadata.gz: 07dbf7e8bb7ac1e7082104ee1d22dcd409b6a6a1d6f899b8c39b1690e1128e40e1e3dbc998d78caa634f464d8b9ce873ba6d519bef004e9cf7e11db9e90d098d
-  data.tar.gz: 5c97353fa00bb68b561cffbb50af26b7229083afc51951a749471d1871932d13a4fb2d89a54bdac8dadbc6704ff09eab17e5ae1bf9d013f66ca8c08a9182c8e8
+  metadata.gz: ba913d760e6f1ac8bac01f68b3ffeb61811952d13a7862731c517dea0ec3e8938b8fb5194113e356fe8a35a02f4d95b306f327772e21f8f6098349bd1235dc53
+  data.tar.gz: 7418c2db5be34b59e123395ee3edf519d703b6e0488a564a85c24dc22344d67e894ad8bb378f88a3cd6e0b9085602be1850dab60c168816e92b081364b720594

data/README.md

@@ -145,6 +145,56 @@ When receiving a `QUIT` signal, if workers still have jobs in-flight, these will
 
 If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats, which will release any claimed jobs back to their queues. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
 
+
+### Dedicated database configuration
+
+Solid Queue can be configured to run on a different database than the main application.
+
+Configure the `connects_to` option in `config/application.rb` or your environment config, with the custom database configuration that will be used in the abstract `SolidQueue::Record` Active Record model.
+
+```ruby
+  # Use a separate DB for Solid Queue
+  config.solid_queue.connects_to = { database: { writing: :solid_queue_primary, reading: :solid_queue_replica } }
+  ```
+
+Add the dedicated database configuration to `config/database.yml`, differentiating between the main app's database and the dedicated `solid_queue` database. Make sure to include the `migrations_paths` for the solid queue database. This is where migration files for Solid Queue tables will reside.
+
+```yml
+default: &default
+  adapter: sqlite3
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
+  timeout: 5000
+
+solid_queue: &solid_queue
+  <<: *default
+  migrations_paths: db/solid_queue_migrate
+
+development:
+  primary:
+    <<: *default
+    # ...
+  solid_queue_primary:
+    <<: *solid_queue
+    # ...
+  solid_queue_replica:
+    <<: *solid_queue
+    # ...
+```
+
+Install migrations and specify the dedicated database name with the `DATABASE` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=solid_queue
+```
+
+Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`), the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
+
+Finally, run the migrations:
+
+```bash
+$ bin/rails db:migrate
+```
+
 ### Other configuration settings
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
 
@@ -156,12 +206,6 @@ There are several settings that control how Solid Queue works that you can set a
   ```ruby
   -> (exception) { Rails.error.report(exception, handled: false) }
   ```
-- `connects_to`: a custom database configuration that will be used in the abstract `SolidQueue::Record` Active Record model. This is required to use a different database than the main app. For example:
-
-  ```ruby
-  # Use a separate DB for Solid Queue
-  config.solid_queue.connects_to = { database: { writing: :solid_queue_primary, reading: :solid_queue_replica } }
-  ```
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
@@ -173,6 +217,10 @@ There are several settings that control how Solid Queue works that you can set a
 - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
 - `enqueue_after_transaction_commit`: whether the job queuing is deferred to after the current Active Record transaction is committed. The default is `false`. [Read more](https://github.com/rails/rails/pull/51426).
 
+## Errors when enqueuing
+Solid Queue will raise a `SolidQueue::Job::EnqueueError` for any Active Record errors that happen when enqueuing a job. The reason for not raising `ActiveJob::EnqueueError` is that this one gets handled by Active Job, causing `perform_later` to return `false` and set `job.enqueue_error`, yielding the job to a block that you need to pass to `perform_later`. This works very well for your own jobs, but makes failure very hard to handle for jobs enqueued by Rails or other gems, such as `Turbo::Streams::BroadcastJob` or `ActiveStorage::AnalyzeJob`, because you don't control the call to `perform_later` in that cases.
+
+In the case of recurring tasks, if such error is raised when enqueuing the job corresponding to the task, it'll be handled and logged but it won't bubble up.
 
 ## Concurrency controls
 Solid Queue extends Active Job with concurrency controls, that allows you to limit how many jobs of a certain type or with certain arguments can run at the same time. When limited in this way, jobs will be blocked from running, and they'll stay blocked until another job finishes and unblocks them, or after the set expiry time (concurrency limit's _duration_) elapses. Jobs are never discarded or lost, only blocked.
@@ -252,7 +300,7 @@ By default, Solid Queue runs in the same DB as your app, and job enqueuing is _n
 If you prefer not to rely on this, or avoid relying on it unintentionally, you should make sure that:
 - You set [`config.active_job.enqueue_after_transaction_commit`](https://edgeguides.rubyonrails.org/configuring.html#config-active-job-enqueue-after-transaction-commit) to `always`, if you're using Rails 7.2+.
 - Or, your jobs relying on specific records are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
-- Or, you configure a database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
+- Or, you configure a different database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
 
   ```ruby
   class ApplicationRecord < ActiveRecord::Base
@@ -288,7 +336,7 @@ Tasks are enqueued at their corresponding times by the dispatcher that owns them
 
 It's possible to run multiple dispatchers with the same `recurring_tasks` configuration. To avoid enqueuing duplicate tasks at the same time, an entry in a new `solid_queue_recurring_executions` table is created in the same transaction as the job is enqueued. This table has a unique index on `task_key` and `run_at`, ensuring only one entry per task per time will be created. This only works if you have `preserve_finished_jobs` set to `true` (the default), and the guarantee applies as long as you keep the jobs around.
 
-Finally, it's possible to configure jobs that aren't handled by Solid Queue. That's it, you can a have a job like this in your app:
+Finally, it's possible to configure jobs that aren't handled by Solid Queue. That is, you can have a job like this in your app:
 ```ruby
 class MyResqueJob < ApplicationJob
   self.queue_adapter = :resque

data/app/models/solid_queue/claimed_execution.rb

@@ -3,6 +3,8 @@
 class SolidQueue::ClaimedExecution < SolidQueue::Execution
   belongs_to :process
 
+  scope :orphaned, -> { where.missing(:process) }
+
   class Result < Struct.new(:success, :error)
     def success?
       success

data/app/models/solid_queue/failed_execution.rb

@@ -21,6 +21,7 @@ module SolidQueue
     def retry
       SolidQueue.instrument(:retry, job_id: job.id) do
         with_lock do
+          job.reset_execution_counters
           job.prepare_for_execution
           destroy!
         end
@@ -32,9 +33,45 @@ module SolidQueue
     end
 
     private
+      JSON_OVERHEAD = 256
+
       def expand_error_details_from_exception
         if exception
-          self.error = { exception_class: exception.class.name, message: exception.message, backtrace: exception.backtrace }
+          self.error = { exception_class: exception_class_name, message: exception_message, backtrace: exception_backtrace }
+        end
+      end
+
+      def exception_class_name
+        exception.class.name
+      end
+
+      def exception_message
+        exception.message
+      end
+
+      def exception_backtrace
+        if (limit = determine_backtrace_size_limit) && exception.backtrace.to_json.bytesize > limit
+          truncate_backtrace(exception.backtrace, limit)
+        else
+          exception.backtrace
+        end
+      end
+
+      def determine_backtrace_size_limit
+        column = self.class.connection.schema_cache.columns_hash(self.class.table_name)["error"]
+        if column.limit.present?
+          column.limit - exception_class_name.bytesize - exception_message.bytesize - JSON_OVERHEAD
+        end
+      end
+
+      def truncate_backtrace(lines, limit)
+        [].tap do |truncated_backtrace|
+          lines.each do |line|
+            if (truncated_backtrace << line).to_json.bytesize > limit
+              truncated_backtrace.pop
+              break
+            end
+          end
         end
       end
   end

data/app/models/solid_queue/job/executable.rb

@@ -6,16 +6,14 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        include ConcurrencyControls, Schedulable
+        include ConcurrencyControls, Schedulable, Retryable
 
         has_one :ready_execution
         has_one :claimed_execution
-        has_one :failed_execution
 
         after_create :prepare_for_execution
 
         scope :finished, -> { where.not(finished_at: nil) }
-        scope :failed, -> { includes(:failed_execution).where.not(failed_execution: { id: nil }) }
       end
 
       class_methods do
@@ -97,18 +95,10 @@ module SolidQueue
         end
       end
 
-      def retry
-        failed_execution&.retry
-      end
-
       def discard
         execution&.discard
       end
 
-      def failed_with(exception)
-        FailedExecution.create_or_find_by!(job_id: id, exception: exception)
-      end
-
       private
         def ready
           ReadyExecution.create_or_find_by!(job_id: id)

data/app/models/solid_queue/job/retryable.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Job
+    module Retryable
+      extend ActiveSupport::Concern
+
+      included do
+        has_one :failed_execution
+
+        scope :failed, -> { includes(:failed_execution).where.not(failed_execution: { id: nil }) }
+      end
+
+      def retry
+        failed_execution&.retry
+      end
+
+      def failed_with(exception)
+        FailedExecution.create_or_find_by!(job_id: id, exception: exception)
+      end
+
+      def reset_execution_counters
+        arguments["executions"] = 0
+        arguments["exception_executions"] = {}
+        save!
+      end
+    end
+  end
+end

data/app/models/solid_queue/job.rb

@@ -2,6 +2,8 @@
 
 module SolidQueue
   class Job < Record
+    class EnqueueError < StandardError; end
+
     include Executable, Clearable, Recurrable
 
     serialize :arguments, coder: JSON
@@ -37,6 +39,11 @@ module SolidQueue
 
         def create_from_active_job(active_job)
           create!(**attributes_from_active_job(active_job))
+        rescue ActiveRecord::ActiveRecordError => e
+          enqueue_error = EnqueueError.new("#{e.class.name}: #{e.message}").tap do |error|
+            error.set_backtrace e.backtrace
+          end
+          raise enqueue_error
         end
 
         def create_all_from_active_jobs(active_jobs)

data/app/models/solid_queue/ready_execution.rb

@@ -24,20 +24,21 @@ module SolidQueue
           return [] if limit <= 0
 
           transaction do
-            job_ids = select_candidates(queue_relation, limit)
-            lock_candidates(job_ids, process_id)
+            candidates = select_candidates(queue_relation, limit)
+            lock_candidates(candidates, process_id)
           end
         end
 
         def select_candidates(queue_relation, limit)
-          queue_relation.ordered.limit(limit).non_blocking_lock.pluck(:job_id)
+          queue_relation.ordered.limit(limit).non_blocking_lock.select(:id, :job_id)
         end
 
-        def lock_candidates(job_ids, process_id)
-          return [] if job_ids.none?
+        def lock_candidates(executions, process_id)
+          return [] if executions.none?
 
-          SolidQueue::ClaimedExecution.claiming(job_ids, process_id) do |claimed|
-            where(job_id: claimed.pluck(:job_id)).delete_all
+          SolidQueue::ClaimedExecution.claiming(executions.map(&:job_id), process_id) do |claimed|
+            ids_to_delete = executions.index_by(&:job_id).values_at(*claimed.map(&:job_id)).map(&:id)
+            where(id: ids_to_delete).delete_all
           end
         end
 

data/app/models/solid_queue/recurring_execution.rb

@@ -2,17 +2,21 @@
 
 module SolidQueue
   class RecurringExecution < Execution
+    class AlreadyRecorded < StandardError; end
+
     scope :clearable, -> { where.missing(:job) }
 
     class << self
       def record(task_key, run_at, &block)
         transaction do
           block.call.tap do |active_job|
-            create!(job_id: active_job.provider_job_id, task_key: task_key, run_at: run_at)
+            if active_job
+              create!(job_id: active_job.provider_job_id, task_key: task_key, run_at: run_at)
+            end
           end
         end
-      rescue ActiveRecord::RecordNotUnique
-        # Task already dispatched
+      rescue ActiveRecord::RecordNotUnique => e
+        raise AlreadyRecorded
       end
 
       def clear_in_batches(batch_size: 500)

data/lib/puma/plugin/solid_queue.rb

@@ -7,15 +7,15 @@ Puma::Plugin.create do
     @log_writer = launcher.log_writer
     @puma_pid = $$
 
+    in_background do
+      monitor_solid_queue
+    end
+
     launcher.events.on_booted do
       @solid_queue_pid = fork do
         Thread.new { monitor_puma }
         SolidQueue::Supervisor.start(mode: :all)
       end
-
-      in_background do
-        monitor_solid_queue
-      end
     end
 
     launcher.events.on_stopped { stop_solid_queue }
@@ -51,12 +51,18 @@ Puma::Plugin.create do
     end
 
     def solid_queue_dead?
-      Process.waitpid(solid_queue_pid, Process::WNOHANG)
+      if solid_queue_started?
+        Process.waitpid(solid_queue_pid, Process::WNOHANG)
+      end
       false
     rescue Errno::ECHILD, Errno::ESRCH
       true
     end
 
+    def solid_queue_started?
+      solid_queue_pid.present?
+    end
+
     def puma_dead?
       Process.ppid != puma_pid
     end

data/lib/solid_queue/dispatcher/recurring_schedule.rb

@@ -11,6 +11,10 @@ module SolidQueue
       @scheduled_tasks = Concurrent::Hash.new
     end
 
+    def empty?
+      configured_tasks.empty?
+    end
+
     def load_tasks
       configured_tasks.each do |task|
         load_task(task)

data/lib/solid_queue/dispatcher/recurring_task.rb

@@ -31,15 +31,23 @@ module SolidQueue
 
     def enqueue(at:)
       SolidQueue.instrument(:enqueue_recurring_task, task: key, at: at) do |payload|
-        if using_solid_queue_adapter?
+        active_job = if using_solid_queue_adapter?
           perform_later_and_record(run_at: at)
         else
           payload[:other_adapter] = true
 
-          perform_later
-        end.tap do |active_job|
-          payload[:active_job_id] = active_job&.job_id
+          perform_later do |job|
+            unless job.successfully_enqueued?
+              payload[:enqueue_error] = job.enqueue_error&.message
+            end
+          end
         end
+
+        payload[:active_job_id] = active_job.job_id if active_job
+      rescue RecurringExecution::AlreadyRecorded
+        payload[:skipped] = true
+      rescue Job::EnqueueError => error
+        payload[:enqueue_error] = error.message
       end
     end
 
@@ -68,8 +76,8 @@ module SolidQueue
         RecurringExecution.record(key, run_at) { perform_later }
       end
 
-      def perform_later
-        job_class.perform_later(*arguments_with_kwargs)
+      def perform_later(&block)
+        job_class.perform_later(*arguments_with_kwargs, &block)
       end
 
       def arguments_with_kwargs

data/lib/solid_queue/dispatcher.rb

@@ -51,6 +51,10 @@ module SolidQueue
         recurring_schedule.unload_tasks
       end
 
+      def all_work_completed?
+        SolidQueue::ScheduledExecution.none? && recurring_schedule.empty?
+      end
+
       def set_procline
         procline "waiting"
       end

data/lib/solid_queue/log_subscriber.rb

@@ -40,12 +40,18 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
   end
 
   def enqueue_recurring_task(event)
-    attributes = event.payload.slice(:task, :at, :active_job_id)
+    attributes = event.payload.slice(:task, :active_job_id, :enqueue_error, :at)
 
     if event.payload[:other_adapter]
-      debug formatted_event(event, action: "Enqueued recurring task outside Solid Queue", **attributes)
+      action = attributes[:active_job_id].present? ? "Enqueued recurring task outside Solid Queue" : "Error enqueuing recurring task"
+      info formatted_event(event, action: action, **attributes)
     else
-      action = attributes[:active_job_id].present? ? "Enqueued recurring task" : "Skipped recurring task – already dispatched"
+      action = case
+      when event.payload[:skipped].present? then "Skipped recurring task – already dispatched"
+      when attributes[:active_job_id].nil?  then "Error enqueuing recurring task"
+      else                                       "Enqueued recurring task"
+      end
+
       info formatted_event(event, action: action, **attributes)
     end
   end

data/lib/solid_queue/processes/poller.rb

@@ -44,7 +44,7 @@ module SolidQueue::Processes
       end
 
       def with_polling_volume
-        if SolidQueue.silence_polling?
+        if SolidQueue.silence_polling? && ActiveRecord::Base.logger
           ActiveRecord::Base.logger.silence { yield }
         else
           yield

data/lib/solid_queue/supervisor.rb

@@ -22,7 +22,8 @@ module SolidQueue
       run_callbacks(:boot) { boot }
 
       start_forks
-      launch_process_prune
+      launch_maintenance_task
+
       supervise
     rescue Processes::GracefulTerminationRequested
       graceful_termination
@@ -65,9 +66,12 @@ module SolidQueue
         configured_processes.each { |configured_process| start_fork(configured_process) }
       end
 
-      def launch_process_prune
-        @prune_task = Concurrent::TimerTask.new(run_now: true, execution_interval: SolidQueue.process_alive_threshold) { prune_dead_processes }
-        @prune_task.execute
+      def launch_maintenance_task
+        @maintenance_task = Concurrent::TimerTask.new(run_now: true, execution_interval: SolidQueue.process_alive_threshold) do
+          prune_dead_processes
+          release_orphaned_executions
+        end
+        @maintenance_task.execute
       end
 
       def shutdown
@@ -106,7 +110,7 @@ module SolidQueue
       end
 
       def stop_process_prune
-        @prune_task&.shutdown
+        @maintenance_task&.shutdown
       end
 
       def delete_pidfile
@@ -117,6 +121,10 @@ module SolidQueue
         wrap_in_app_executor { SolidQueue::Process.prune }
       end
 
+      def release_orphaned_executions
+        wrap_in_app_executor { SolidQueue::ClaimedExecution.orphaned.release_all }
+      end
+
       def start_fork(configured_process)
         configured_process.supervised_by process
 

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.3.2"
+  VERSION = "0.3.4"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.3.4
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-05-30 00:00:00.000000000 Z
+date: 2024-07-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -199,6 +199,7 @@ files:
 - app/models/solid_queue/job/concurrency_controls.rb
 - app/models/solid_queue/job/executable.rb
 - app/models/solid_queue/job/recurrable.rb
+- app/models/solid_queue/job/retryable.rb
 - app/models/solid_queue/job/schedulable.rb
 - app/models/solid_queue/pause.rb
 - app/models/solid_queue/process.rb
@@ -265,7 +266,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.9
+rubygems_version: 3.5.16
 signing_key:
 specification_version: 4
 summary: Database-backed Active Job backend.