solid_queue 0.7.0 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 23836c755cda6dd9bec594148ad92ea19740da4125e9115d17c44bd2543d752c
-  data.tar.gz: 40d91182c13f155a86879f04abcf4232a50f3502f4038b5e9e07a3c15da8def2
+  metadata.gz: a7234dc4430998648196bf2d3905b66bb85e265c2393121a7c5343fed36b1996
+  data.tar.gz: 216a0918e29194e6d11fe4bf365183228127f8390658a11ace329523ad41468c
 SHA512:
-  metadata.gz: f8ac683f1a0e635762d0d93b9b74f712c5fc5465381ea51230448dc2f176de67aafb0ee7031ae9e9cfee73edcee7bd70352e78aefd49ae25ea71a7485af95aff
-  data.tar.gz: b02eac304a30fb8c7e9e2627446a31bb56d22740e863b1d2749838429416f886da78d19d2000b533e3672576387ee075b8efb3c330f6177b5cd76a27955cd433
+  metadata.gz: a080aedf20f39940d8c25e8a14628811801b8fd4832fbcb926beecd0d1ce4c694ec8d80b608656f9de80cc8cc69b525f62d4fe7bc7258408aa6d6af62292d4fd
+  data.tar.gz: 66d24c1bb1c7cb1b9afbcf8a0b667df624dd3f47cea92887fbcd69d93b1832af3c44afb570d1cfb459c37e2b3c163900e0dd2febb067ee9179b4c49514e4b359

data/README.md

@@ -208,17 +208,47 @@ Finally, run the migrations:
 $ bin/rails db:migrate
 ```
 
+## Lifecycle hooks
+
+In Solid queue, you can hook into two different points in the supervisor's life:
+- `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
+- `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
+
+And into two different points in a worker's life:
+- `worker_start`: after the worker has finished booting and right before it starts the polling loop.
+- `worker_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
+
+You can use the following methods with a block to do this:
+```ruby
+SolidQueue.on_start
+SolidQueue.on_stop
+
+SolidQueue.on_worker_start
+SolidQueue.on_worker_stop
+```
+
+For example:
+```ruby
+SolidQueue.on_start { start_metrics_server }
+SolidQueue.on_stop { stop_metrics_server }
+```
+
+These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
+
+
 ### Other configuration settings
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
 
 There are several settings that control how Solid Queue works that you can set as well:
 - `logger`: the logger you want Solid Queue to use. Defaults to the app logger.
 - `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap asynchronous operations, defaults to the app executor
-- `on_thread_error`: custom lambda/Proc to call when there's an error within a thread that takes the exception raised as argument. Defaults to
+- `on_thread_error`: custom lambda/Proc to call when there's an error within a Solid Queue thread that takes the exception raised as argument. Defaults to
 
   ```ruby
   -> (exception) { Rails.error.report(exception, handled: false) }
   ```
+  **This is not used for errors raised within a job execution**. Errors happening in jobs are handled by Active Job's `retry_on` or `discard_on`, and ultimately will result in [failed jobs](#failed-jobs-and-retries). This is for errors happening within Solid Queue itself.
+
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
@@ -283,6 +313,8 @@ In this case, if we have a `Box::MovePostingsByContactToDesignatedBoxJob` job en
 
 Note that the `duration` setting depends indirectly on the value for `concurrency_maintenance_interval` that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked. In general, you should set `duration` in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.
 
+Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
+
 Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for gaining the lock, and whenever they get it, they'll be run. It doesn't matter if they had gained the lock already in the past.
 
 ## Failed jobs and retries

data/lib/solid_queue/lifecycle_hooks.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module LifecycleHooks
+    extend ActiveSupport::Concern
+
+    included do
+      mattr_reader :lifecycle_hooks, default: { start: [], stop: [] }
+    end
+
+    class_methods do
+      def on_start(&block)
+        self.lifecycle_hooks[:start] << block
+      end
+
+      def on_stop(&block)
+        self.lifecycle_hooks[:stop] << block
+      end
+
+      def clear_hooks
+        self.lifecycle_hooks[:start] = []
+        self.lifecycle_hooks[:stop] = []
+      end
+    end
+
+    private
+      def run_start_hooks
+        run_hooks_for :start
+      end
+
+      def run_stop_hooks
+        run_hooks_for :stop
+      end
+
+      def run_hooks_for(event)
+        self.class.lifecycle_hooks.fetch(event, []).each do |block|
+          block.call
+        rescue Exception => exception
+          handle_thread_error(exception)
+        end
+      end
+  end
+end

data/lib/solid_queue/processes/runnable.rb

@@ -7,11 +7,7 @@ module SolidQueue::Processes
     attr_writer :mode
 
     def start
-      @stopped = false
-
-      SolidQueue.instrument(:start_process, process: self) do
-        run_callbacks(:boot) { boot }
-      end
+      boot
 
       if running_async?
         @thread = create_thread { run }
@@ -25,10 +21,6 @@ module SolidQueue::Processes
       @thread&.join
     end
 
-    def alive?
-      !running_async? || @thread.alive?
-    end
-
     private
       DEFAULT_MODE = :async
 
@@ -37,9 +29,15 @@ module SolidQueue::Processes
       end
 
       def boot
-        if running_as_fork?
-          register_signal_handlers
-          set_procline
+        SolidQueue.instrument(:start_process, process: self) do
+          run_callbacks(:boot) do
+            @stopped = false
+
+            if running_as_fork?
+              register_signal_handlers
+              set_procline
+            end
+          end
         end
       end
 

data/lib/solid_queue/supervisor.rb

@@ -2,6 +2,7 @@
 
 module SolidQueue
   class Supervisor < Processes::Base
+    include LifecycleHooks
     include Maintenance, Signals, Pidfiled
 
     class << self
@@ -27,6 +28,7 @@ module SolidQueue
 
     def start
       boot
+      run_start_hooks
 
       start_processes
       launch_maintenance_task
@@ -36,6 +38,7 @@ module SolidQueue
 
     def stop
       @stopped = true
+      run_stop_hooks
     end
 
     private

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.7.0"
+  VERSION = "0.7.1"
 end

data/lib/solid_queue/worker.rb

@@ -2,6 +2,11 @@
 
 module SolidQueue
   class Worker < Processes::Poller
+    include LifecycleHooks
+
+    after_boot :run_start_hooks
+    before_shutdown :run_stop_hooks
+
     attr_accessor :queues, :pool
 
     def initialize(**options)

data/lib/solid_queue.rb

@@ -43,6 +43,16 @@ module SolidQueue
   mattr_accessor :clear_finished_jobs_after, default: 1.day
   mattr_accessor :default_concurrency_control_period, default: 3.minutes
 
+  delegate :on_start, :on_stop, to: Supervisor
+
+  def on_worker_start(...)
+    Worker.on_start(...)
+  end
+
+  def on_worker_stop(...)
+    Worker.on_stop(...)
+  end
+
   def supervisor?
     supervisor
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.7.1
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-09-02 00:00:00.000000000 Z
+date: 2024-09-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -253,6 +253,7 @@ files:
 - lib/solid_queue/dispatcher/concurrency_maintenance.rb
 - lib/solid_queue/dispatcher/recurring_schedule.rb
 - lib/solid_queue/engine.rb
+- lib/solid_queue/lifecycle_hooks.rb
 - lib/solid_queue/log_subscriber.rb
 - lib/solid_queue/pool.rb
 - lib/solid_queue/processes/base.rb