patient_http-solid_queue 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2db0a06a8b31785cab9595f85351bef93277ba420f11c21b3858e24c9fbf7c18
-  data.tar.gz: 73facc946369816cf6e9384b14f5300420446c2bd4d93e91a2b10ad5f6620d64
+  metadata.gz: c5b556f667fd07c4e9dc41089d036b95fbe0c6ef9e820d5dfde5aea7f74fd76e
+  data.tar.gz: eb4583469ba5e2db21d19af63e5f6b601c2bff969e9c116e44087a1cea2717cb
 SHA512:
-  metadata.gz: 86859b38cba5d753001c093f32783905944852acc249a5c9dd7b52da6940dc8c23458e75fe257767f7e07355aec2be4788d27c4cdae62a87a7eba5eb79827728
-  data.tar.gz: 6d0444ccb2cfb9ee06e52423d84ab0aeca15bf8974a23b02a7c196ccfca99ef047ea28fddd7e864324a5428725fa7677168ca98b9f5b8ad179636250efa9876c
+  metadata.gz: a92d7a43aed5ed18160344e165c620dad2c2aae02184a1c4b3be41c59b3a41a5cc16c8628ff44e55c38e9fe0e4b5a24b8cde5aeb572db4792eca34de0c736eea
+  data.tar.gz: 77874d19a4dcbfbe0f6657b780d30a33a321eec9f6fad9cc5689a746bdfe1287bfdc28e33a64e4ca10610b19bdbcd0665dd515f8c264be440da21f51790f4d62

data/CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.2.0
+
+### Added
+
+- Named processors: declare profiles with `config.processor(:llm, max_connections: 200)` and route requests with `PatientHttp::SolidQueue.execute(request, processor: :llm)` or a `processor:` option on the request itself. Each profile runs as an independent processor with its own capacity, timeouts, and threads, so one workload class cannot starve another. The processor name is serialized into the job arguments, so retries and crash recovery keep their routing. A job that names an unconfigured processor raises `PatientHttp::UnknownProcessorError` and is retried, which makes new profile names safe to roll out gradually. Jobs from older gem versions run on the `:default` processor.
+- Capacity fast path: requests are rejected with a cheap in-memory capacity check (`Processor#capacity_available?`) before the durable registry record is written, so a full processor rejects without a database insert and delete.
+- The `completion_failed` processor event is handled by keeping the crash-recovery registry entry and releasing it from this process, so a request whose result could not be delivered is re-enqueued by the orphan collector on its next pass instead of being silently lost.
+- A failure to write the crash-recovery registry entry raises `PatientHttp::SolidQueue::RegistrationError`, which `RequestJob` retries with backoff so a transient database issue does not fail the request outright.
+
+### Changed
+
+- Requests are now registered in the crash-recovery registry when the processor accepts them (`request_enqueued`, on the job worker thread), instead of when they start processing (`request_start`, on the reactor thread). The registry insert no longer blocks the event loop, queued requests are covered by heartbeats, and a failure to write the record rejects the request and raises to the caller so a task is never accepted without a durable record. Rejected and re-enqueued tasks now remove their registry entries, closing a duplicate-delivery window after process restarts.
+- With patient_http 1.5.0, result delivery (response decoding, callback job enqueueing, registry cleanup) runs on the processor's completion worker threads instead of the reactor thread. Size the database pool to cover `completion_threads` plus the monitor thread.
+- TaskMonitor operations check out a database connection only for the duration of each operation, so gem-owned threads do not pin connections from the pool.
+- The task monitor thread is now owned by the module and shared across all processors; `ProcessorObserver.new` takes a `task_monitor:` keyword argument.
+- The `patient_http` dependency floor is now 1.5.0.
+
 ## 1.1.0
 
 ### Changed

data/README.md

@@ -230,6 +230,34 @@ PatientHttp.execute(request: request, callback: MyCallback, callback_args: {user
 
 See the [patient_http docs](https://github.com/bdurand/patient_http) for the full `Request` and `Response` API reference.
 
+### Named Processors
+
+By default all requests share one processor and one `max_connections` cap. When one process serves workload classes with very different profiles (for example, large slow API calls and small fast webhook deliveries), a burst of one class can consume all of the capacity the other class needs. Named processor profiles isolate them:
+
+```ruby
+PatientHttp::SolidQueue.configure do |config|
+  config.processor(:llm, max_connections: 200, request_timeout: 120)
+  config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+end
+```
+
+Each profile runs as an independent processor in the process, with its own capacity, timeouts, and threads. Profile options override the top-level configuration; anything not overridden (secrets, preprocessors, payload stores, encryption, logger) is shared. The `:default` processor always exists; declare `config.processor(:default, ...)` to override its options.
+
+Route a request to a processor in any of these ways:
+
+```ruby
+# Explicit option on execute
+PatientHttp::SolidQueue.execute(request, callback: MyCallback, processor: :llm)
+
+# On the request itself (survives serialization, retries, and crash recovery)
+request = PatientHttp::Request.new(:get, url, processor: :llm)
+
+# Through a request template
+template = PatientHttp::RequestTemplate.new(base_url: url, processor: :llm)
+```
+
+The processor name is serialized into the job arguments, so Active Job retries and crash recovery keep their routing. A job that names a processor that is not configured in the executing process raises `PatientHttp::UnknownProcessorError` and is retried with backoff; this makes new profile names safe to roll out gradually. Jobs enqueued by older gem versions run on the `:default` processor.
+
 ### Using Request Templates
 
 For repeated requests to the same API, use `PatientHttp::RequestTemplate` to share configuration:
@@ -419,6 +447,16 @@ PatientHttp::SolidQueue.configure do |config|
   # Queue name for RequestJob and CallbackJob (default: nil, Active Job default)
   config.queue_name = "async_http"
 
+  # Number of threads that decode responses and deliver results (default: 2)
+  config.completion_threads = 2
+
+  # Maximum connections per host (default: nil, unlimited)
+  config.max_connections_per_host = 32
+
+  # Named processor profiles for workload isolation (see Named Processors)
+  config.processor(:llm, max_connections: 200, request_timeout: 120)
+  config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+
   # Custom logger (defaults to SolidQueue.logger)
   config.logger = Rails.logger
 
@@ -440,6 +478,9 @@ See the [Configuration](lib/patient_http/solid_queue/configuration.rb) class for
 - `retries`: Number of times to retry a failed request before calling the error callback.
 - `max_response_size`: Set this to limit the maximum size of HTTP responses. This helps prevent excessive memory usage from unexpectedly large responses. Responses need to be serialized as Active Job arguments and very large responses may cause performance issues. If a response body is text content, it will be compressed to save space. However, binary content needs to be Base64 encoded which increases size by ~33%.
 - `payload_store_threshold`: Lower this if your queue backend struggles with large payloads; higher values avoid extra external storage reads/writes.
+- `max_connections_per_host`: Bounds sockets per host. Verify the process file descriptor limit covers `max_connections` plus pooled idle host connections plus the application's own connections; raise the limit if needed.
+- `completion_threads`: Number of threads that decode responses and deliver results (default 2). Increase when result callbacks do heavier work and completions back up behind them. Size the Active Record connection pool to cover these threads plus the task monitor thread in addition to the worker threads.
+- `shutdown_timeout`: Must be below the process supervisor's termination window so the drain finishes before a hard kill. The default derives it from Solid Queue's own shutdown timeout; check any additional supervisor stop timeout as well.
 - `heartbeat_interval` and `orphan_threshold`: For high-churn workloads, keep `heartbeat_interval` as large as your recovery SLO allows (while still less than `orphan_threshold`) to reduce write/update pressure on monitoring tables. If Solid Queue uses PostgreSQL and request volume is high, tune autovacuum for the queue database tables because `inflight_requests` is intentionally insert/update/delete heavy.
 
 > [!IMPORTANT]

data/VERSION

@@ -1 +1 @@
-1.1.0
+1.2.0

data/lib/patient_http/solid_queue/configuration.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "delegate"
+
 module PatientHttp
   module SolidQueue
     # Configuration for the Solid Queue Async HTTP gem.
@@ -64,6 +66,7 @@ module PatientHttp
 
         super(**pool_options)
 
+        @processor_profiles = {default: {}}
         self.queue_name = queue_name
         self.heartbeat_interval = heartbeat_interval
         self.orphan_threshold = orphan_threshold
@@ -71,6 +74,64 @@ module PatientHttp
         self.on_retries_exhausted = on_retries_exhausted
       end
 
+      # Declare a named processor profile.
+      #
+      # Each profile becomes an independent processor with its own capacity,
+      # timeouts, and threads. Options are overrides applied on top of this
+      # configuration's HTTP pool options. Calling this with no options
+      # declares a profile that inherits every option. Requests select a
+      # processor with the +processor:+ option on
+      # +PatientHttp::SolidQueue.execute+ (or on the request itself). The
+      # +:default+ profile always exists; declaring it overrides options for
+      # the default processor.
+      #
+      # @example
+      #   PatientHttp::SolidQueue.configure do |config|
+      #     config.processor(:llm, max_connections: 200, request_timeout: 120)
+      #     config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+      #   end
+      #
+      # @param name [Symbol, String] the processor name
+      # @param options [Hash] overrides for PatientHttp::Configuration options
+      # @return [Hash] the stored options for the profile
+      def processor(name, **options)
+        key = normalize_processor_name(name)
+        @processor_profiles[key] = normalize_profile_options!(options)
+      end
+
+      # Read back the options declared for a named processor profile.
+      #
+      # @param name [Symbol, String] the processor name
+      # @return [Hash, nil] the stored options, or nil if the profile is not declared
+      def processor_options(name)
+        @processor_profiles[normalize_processor_name(name)]
+      end
+
+      # All declared processor profiles. Always includes :default.
+      #
+      # @return [Hash{Symbol => Hash}] profile options by processor name
+      def processor_profiles
+        @processor_profiles.dup
+      end
+
+      # Build the effective configuration for a named processor. The default
+      # profile with no overrides is this configuration itself; other profiles
+      # get a view of this configuration with their overrides applied, so they
+      # share secrets, preprocessors, payload stores, and encryption.
+      #
+      # @param name [Symbol, String] the processor name
+      # @return [PatientHttp::Configuration] the configuration for the processor
+      # @raise [ArgumentError] if the profile is not declared
+      def processor_config(name)
+        key = normalize_processor_name(name)
+        profile = @processor_profiles[key]
+        raise ArgumentError.new("Unknown processor profile: #{name.inspect}") unless profile
+
+        return self if profile.empty?
+
+        ProfileConfiguration.new(self, profile)
+      end
+
       def payload_store_threshold=(value)
         validate_positive_integer(:payload_store_threshold, value)
         @payload_store_threshold = value
@@ -121,10 +182,24 @@ module PatientHttp
           "heartbeat_interval" => heartbeat_interval,
           "orphan_threshold" => orphan_threshold,
           "queue_name" => queue_name,
-          "on_retries_exhausted" => on_retries_exhausted ? "defined" : nil
+          "on_retries_exhausted" => on_retries_exhausted ? "defined" : nil,
+          "processor_profiles" => processor_profiles.keys.map(&:to_s)
         )
       end
 
+      # View of a base configuration with a profile's option overrides applied.
+      # Everything not overridden (secrets, preprocessors, payload stores,
+      # encryption, logging) delegates to the base configuration, so all
+      # processors share those registries.
+      class ProfileConfiguration < SimpleDelegator
+        def initialize(base_configuration, overrides)
+          super(base_configuration)
+          overrides.each do |key, value|
+            define_singleton_method(key) { value }
+          end
+        end
+      end
+
       private
 
       def apply_queue_name(name)
@@ -132,6 +207,28 @@ module PatientHttp
         PatientHttp::SolidQueue::CallbackJob.queue_as(name)
       end
 
+      # Profile options must be valid PatientHttp::Configuration options. A
+      # throwaway configuration exercises each option's own validation and
+      # normalization, so the stored value is what the writer would have
+      # produced rather than the raw input.
+      def normalize_profile_options!(options)
+        return options if options.empty?
+
+        probe = PatientHttp::Configuration.new(**options)
+        options.to_h do |key, value|
+          [key, probe.respond_to?(key) ? probe.public_send(key) : value]
+        end
+      rescue ArgumentError => e
+        raise ArgumentError.new("Invalid processor profile options: #{e.message}")
+      end
+
+      def normalize_processor_name(name)
+        key = name.to_s
+        raise ArgumentError.new("processor name cannot be empty") if key.empty?
+
+        key.to_sym
+      end
+
       def validate_heartbeat_and_threshold
         return unless @heartbeat_interval && @orphan_threshold
         return unless @heartbeat_interval >= @orphan_threshold

data/lib/patient_http/solid_queue/processor_observer.rb

@@ -2,37 +2,68 @@
 
 module PatientHttp
   module SolidQueue
-    # Processor Observer that monitors for crashed processes in order to
-    # re-enqueue workers, and manages the TaskMonitor lifecycle.
+    # Processor observer that maintains the crash-recovery registry for one
+    # processor. The task monitor is shared across all processors in the
+    # process; the module owns it and the monitor thread.
+    #
+    # Tasks are registered in the crash-recovery registry when the processor
+    # accepts them, before Processor#enqueue returns, so a request always has
+    # a durable record from the moment the caller hands it off. Registration
+    # runs on the caller's thread (a job worker thread), not the reactor
+    # thread. The entry is removed when the request completes or when an
+    # Active Job owns the request again (the task was rejected or
+    # re-enqueued). When result delivery fails (completion_failed), the entry
+    # is deliberately kept so the orphan collector re-enqueues the request
+    # instead of losing it.
     class ProcessorObserver < PatientHttp::ProcessorObserver
       attr_reader :task_monitor
 
-      def initialize(processor)
+      def initialize(processor, task_monitor:)
         @processor = processor
-        @task_monitor = TaskMonitor.new(processor.config)
-        @monitor_thread = TaskMonitorThread.new(
-          processor.config,
-          @task_monitor,
-          -> { @processor.inflight_request_ids }
-        )
+        @task_monitor = task_monitor
+        @requeued_task_ids = Set.new
+        @requeued_mutex = Mutex.new
       end
 
-      def start
-        @monitor_thread.start
+      def request_enqueued(request_task)
+        task_monitor.register(request_task)
       end
 
-      def stop
-        @monitor_thread.stop
-        task_monitor.remove_process
+      def request_rejected(request_task)
+        task_monitor.unregister(request_task)
       end
 
-      def request_start(request_task)
-        task_monitor.register(request_task)
+      def request_requeued(request_task)
+        task_monitor.unregister(request_task)
+        # The re-enqueue path fires request_end after request_requeued, but
+        # only for tasks that already started. Remember those tasks so that
+        # request_end does not unregister a second time. A task that never
+        # started gets no request_end, so remembering it would leak the id
+        # forever.
+        return unless request_task.started?
+
+        @requeued_mutex.synchronize { @requeued_task_ids << request_task.id }
       end
 
       def request_end(request_task)
+        requeued = @requeued_mutex.synchronize { @requeued_task_ids.delete?(request_task.id) }
+        return if requeued
+
         task_monitor.unregister(request_task)
       end
+
+      def completion_failed(request_task, error)
+        # Keep the crash-recovery registry entry, but hand it off to the orphan
+        # collector. Orphan collection ignores records that belong to a live
+        # process, so the entry has to be released for the request to be
+        # re-enqueued on the next pass rather than on the next process restart.
+        task_monitor.release(request_task)
+
+        PatientHttp::SolidQueue.configuration.logger&.error(
+          "[PatientHttp::SolidQueue] Result delivery failed for request #{request_task.id}; " \
+          "leaving crash-recovery record for re-enqueue: #{error.class} - #{error.message}"
+        )
+      end
     end
   end
 end

data/lib/patient_http/solid_queue/request_executor.rb

@@ -14,6 +14,8 @@ module PatientHttp
         # @param callback_args [#to_h, nil] Arguments to pass to callback
         # @param raise_error_responses [Boolean] If true, treats non-2xx responses as errors
         # @param request_id [String, nil] Unique request ID for tracking
+        # @param processor_name [Symbol, String, nil] Name of the processor profile to run
+        #   the request on. Defaults to the request's own processor name or :default.
         # @return [String] the request ID
         # @api private
         def execute(
@@ -23,12 +25,19 @@ module PatientHttp
           synchronous: false,
           callback_args: nil,
           raise_error_responses: false,
-          request_id: nil
+          request_id: nil,
+          processor_name: nil
         )
           active_job_data = validate_active_job_data(active_job_data)
           task_handler = TaskHandler.new(active_job_data)
           config = PatientHttp::SolidQueue.configuration
 
+          # Resolve the processor profile up front so the task is built with
+          # the options of the processor that will run it. An unknown name
+          # falls back to the base configuration here and is reported below.
+          name = (processor_name || request.processor || :default).to_sym
+          profile_config = config.processor_profiles.key?(name) ? config.processor_config(name) : config
+
           task = PatientHttp::RequestTask.new(
             request: request,
             task_handler: task_handler,
@@ -36,24 +45,43 @@ module PatientHttp
             callback_args: callback_args,
             raise_error_responses: raise_error_responses,
             id: request_id,
-            default_max_redirects: config.max_redirects
+            default_max_redirects: profile_config.max_redirects
           )
 
           if synchronous || async_disabled?
             PatientHttp::SynchronousExecutor.new(
               task,
-              config: config,
+              config: profile_config,
               on_complete: ->(response) { PatientHttp::SolidQueue.invoke_completion_callbacks(response) },
               on_error: ->(error) { PatientHttp::SolidQueue.invoke_error_callbacks(error) }
             ).call
             return task.id
           end
 
-          processor = PatientHttp::SolidQueue.processor
+          # Look up the named processor. An unknown name raises so the job
+          # lands in Active Job's retry mechanism instead of being dropped;
+          # this covers rolling deploys where an old process has not
+          # configured a new profile yet.
+          processor = PatientHttp::SolidQueue.processor(name)
+          if processor.nil? && !config.processor_profiles.key?(name)
+            raise PatientHttp::UnknownProcessorError, "No processor profile configured for #{name.inspect}"
+          end
+
           unless processor&.running?
             raise PatientHttp::NotRunningError, "Cannot enqueue request: processor is not running"
           end
 
+          # Advisory capacity check before enqueueing. A real enqueue writes
+          # the durable registry record before the authoritative capacity
+          # check, so a full processor would pay a database insert and delete
+          # just to be rejected. This peek rejects for free; the race where
+          # capacity fills after the peek falls through to the normal
+          # rejection path.
+          unless processor.capacity_available?
+            raise PatientHttp::MaxCapacityError,
+              "Cannot enqueue request: processor #{name} is at max capacity (#{processor.config.max_connections} connections)"
+          end
+
           processor.enqueue(task)
           task.id
         end

data/lib/patient_http/solid_queue/request_job.rb

@@ -13,7 +13,14 @@ module PatientHttp
       # Rejection due to backpressure is part of normal operation: retry until the
       # processor has capacity again. NotRunningError covers jobs that run during
       # the narrow window when the processor is draining or stopping.
-      retry_on PatientHttp::MaxCapacityError, PatientHttp::NotRunningError, wait: :polynomially_longer, attempts: :unlimited
+      # UnknownProcessorError covers rolling deploys where a job names a processor
+      # profile that an old process has not configured yet.
+      retry_on PatientHttp::MaxCapacityError, PatientHttp::NotRunningError,
+        PatientHttp::UnknownProcessorError, wait: :polynomially_longer, attempts: :unlimited
+
+      # A registry write failure is usually a transient database issue, so give
+      # it a bounded number of retries before the job is marked failed.
+      retry_on PatientHttp::SolidQueue::RegistrationError, wait: :polynomially_longer, attempts: 10
 
       # Capture the Active Job serialized hash into Context so RequestExecutor can use it.
       around_perform do |job, block|
@@ -36,7 +43,9 @@ module PatientHttp
       # @param raise_error_responses [Boolean, nil] Whether to treat non-2xx responses as errors
       # @param callback_args [Hash, nil] Arguments to pass to the callback
       # @param request_id [String, nil] Unique request ID for tracking
-      def perform(data, callback_service_name, raise_error_responses, callback_args, request_id)
+      # @param processor_name [String, nil] Name of the processor profile to run the request
+      #   on; nil (jobs enqueued by older versions) runs on the default processor
+      def perform(data, callback_service_name, raise_error_responses, callback_args, request_id, processor_name = nil)
         actual_data = PatientHttp::ExternalStorage.storage_ref?(data) ? PatientHttp::SolidQueue.external_storage.fetch(data) : data
         actual_data = PatientHttp::SolidQueue.decrypt(actual_data)
 
@@ -53,7 +62,8 @@ module PatientHttp
           raise_error_responses: raise_error_responses,
           callback_args: callback_args,
           active_job_data: active_job_data,
-          request_id: request_id
+          request_id: request_id,
+          processor_name: processor_name || "default"
         )
       end
     end

data/lib/patient_http/solid_queue/task_monitor.rb

@@ -19,8 +19,13 @@ module PatientHttp
       # @return [Configuration] the configuration object
       attr_reader :config
 
-      def initialize(config)
+      # @param config [Configuration] the configuration object
+      # @param max_connections [#call, nil] callable returning the process's total
+      #   configured max connections; defaults to the configuration's value. With
+      #   named processors the module passes a sum across all processors.
+      def initialize(config, max_connections: nil)
         @config = config
+        @max_connections_source = max_connections || -> { config.max_connections }
         hostname = ::Socket.gethostname.force_encoding("UTF-8").tr(":/", "-")
         pid = ::Process.pid
         @lock_identifier = "#{hostname}:#{pid}:#{SecureRandom.hex(8)}".freeze
@@ -28,23 +33,32 @@ module PatientHttp
 
       # Register a request as inflight in the database.
       #
+      # Runs on the caller's thread via the request_enqueued observer event.
+      # Errors propagate so a task is never accepted without a durable record:
+      # the processor rejects the task and the enqueue raises to the caller.
+      # They are wrapped in RegistrationError so the job retries instead of
+      # failing outright, since the usual cause is a transient database issue.
+      #
       # @param task [PatientHttp::RequestTask] the request task to register
+      # @raise [RegistrationError] if the record cannot be written
       # @return [void]
       def register(task)
         job_payload = task.task_handler.active_job_data.to_json
         task_id = full_task_id(task.id)
         now = Time.current
 
-        InflightRequest.create!(
-          task_id: task_id,
-          process_id: @lock_identifier,
-          job_payload: job_payload,
-          heartbeat_at: now,
-          created_at: now
-        )
+        with_connection do
+          InflightRequest.create!(
+            task_id: task_id,
+            process_id: @lock_identifier,
+            job_payload: job_payload,
+            heartbeat_at: now,
+            created_at: now
+          )
+        end
       rescue => e
-        @config.logger&.error("[PatientHttp::SolidQueue] Failed to register task #{task_id}: #{e.message}")
-        raise if PatientHttp.testing?
+        @config.logger&.error("[PatientHttp::SolidQueue] Failed to register task #{task_id}: #{e.class} - #{e.message}")
+        raise RegistrationError.new("Failed to register task #{task_id}: #{e.class} - #{e.message}")
       end
 
       # Unregister a request from the database (called when request completes).
@@ -53,12 +67,36 @@ module PatientHttp
       # @return [void]
       def unregister(task)
         task_id = full_task_id(task.id)
-        InflightRequest.where(task_id: task_id).delete_all
+        with_connection do
+          InflightRequest.where(task_id: task_id).delete_all
+        end
       rescue => e
         @config.logger&.error("[PatientHttp::SolidQueue] Failed to unregister task #{task_id}: #{e.message}")
         raise if PatientHttp.testing?
       end
 
+      # Release a request from this process so the orphan collector re-enqueues
+      # it on its next pass. Used when a result could not be delivered: the
+      # request is no longer tracked here, so its record must not keep looking
+      # like it belongs to a live process. Orphan collection skips records
+      # whose process is still registered, which would otherwise strand the
+      # request until this process exits.
+      #
+      # @param task [PatientHttp::RequestTask] the request task to release
+      # @return [void]
+      def release(task)
+        task_id = full_task_id(task.id)
+        with_connection do
+          InflightRequest.where(task_id: task_id).update_all(
+            process_id: released_process_id,
+            heartbeat_at: Time.at(0).utc
+          )
+        end
+      rescue => e
+        @config.logger&.error("[PatientHttp::SolidQueue] Failed to release task #{task_id}: #{e.message}")
+        raise if PatientHttp.testing?
+      end
+
       # Update heartbeat timestamps for multiple requests in a single operation.
       #
       # @param task_ids [Array<String>] the request IDs to update
@@ -67,7 +105,9 @@ module PatientHttp
         return if task_ids.empty?
 
         full_ids = task_ids.map { |id| full_task_id(id) }
-        InflightRequest.where(task_id: full_ids).update_all(heartbeat_at: Time.current)
+        with_connection do
+          InflightRequest.where(task_id: full_ids).update_all(heartbeat_at: Time.current)
+        end
       rescue => e
         @config.logger&.error("[PatientHttp::SolidQueue] Failed to update heartbeats: #{e.message}")
         raise if PatientHttp.testing?
@@ -77,10 +117,14 @@ module PatientHttp
       #
       # @return [void]
       def ping_process
-        ProcessRegistration.upsert(
-          {process_id: @lock_identifier, max_connections: @config.max_connections, last_seen_at: Time.current},
-          unique_by: upsert_unique_by(:process_id)
-        )
+        max_connections = @max_connections_source.call
+
+        with_connection do
+          ProcessRegistration.upsert(
+            {process_id: @lock_identifier, max_connections: max_connections, last_seen_at: Time.current},
+            unique_by: upsert_unique_by(:process_id)
+          )
+        end
       rescue => e
         @config.logger&.error("[PatientHttp::SolidQueue] Failed to ping process: #{e.message}")
         raise if PatientHttp.testing?
@@ -90,7 +134,9 @@ module PatientHttp
       #
       # @return [void]
       def remove_process
-        ProcessRegistration.where(process_id: @lock_identifier).delete_all
+        with_connection do
+          ProcessRegistration.where(process_id: @lock_identifier).delete_all
+        end
       rescue => e
         @config.logger&.error("[PatientHttp::SolidQueue] Failed to remove process: #{e.message}")
         raise if PatientHttp.testing?
@@ -190,7 +236,9 @@ module PatientHttp
       # @return [Boolean]
       # @api private
       def registered?(task)
-        InflightRequest.where(task_id: full_task_id(task.id)).exists?
+        with_connection do
+          InflightRequest.where(task_id: full_task_id(task.id)).exists?
+        end
       end
 
       # Clear all records. Only allowed in test environment.
@@ -210,6 +258,19 @@ module PatientHttp
 
       private
 
+      # Check out a database connection only for the duration of the work so
+      # gem-owned threads (completion workers, the monitor thread) do not pin
+      # connections from the pool between operations.
+      def with_connection(&block)
+        Record.connection_pool.with_connection(&block)
+      end
+
+      # Process identifier stamped on released records. It is never registered
+      # in the process table, so orphan collection always considers it dead.
+      def released_process_id
+        "#{@lock_identifier}:released"
+      end
+
       def ensure_gc_lock_row!
         GcLock.insert_all([{lock_name: GC_LOCK_NAME}])
       end

data/lib/patient_http/solid_queue/task_monitor_thread.rb

@@ -20,11 +20,12 @@ module PatientHttp
       #
       # @param config [Configuration] the configuration object
       # @param task_monitor [TaskMonitor] the inflight request registry
-      # @param inflight_ids_callback [Proc] callback to get current inflight request IDs
-      def initialize(config, task_monitor, inflight_ids_callback)
+      # @param tracked_ids_callback [Proc] callback to get the IDs of all requests the
+      #   processors are tracking (queued, pending, and in-flight)
+      def initialize(config, task_monitor, tracked_ids_callback)
         @config = config
         @task_monitor = task_monitor
-        @inflight_ids_callback = inflight_ids_callback
+        @tracked_ids_callback = tracked_ids_callback
         @thread = nil
         @running = Concurrent::AtomicBoolean.new(false)
         @stop_signal = Concurrent::Event.new
@@ -112,12 +113,12 @@ module PatientHttp
       end
 
       def update_heartbeats
-        request_ids = @inflight_ids_callback.call
+        request_ids = @tracked_ids_callback.call
         return if request_ids.empty?
 
         @task_monitor.update_heartbeats(request_ids)
 
-        @config.logger&.debug("[PatientHttp::SolidQueue] Updated heartbeats for #{request_ids.size} inflight requests")
+        @config.logger&.debug("[PatientHttp::SolidQueue] Updated heartbeats for #{request_ids.size} tracked requests")
       rescue => e
         @config.logger&.error("[PatientHttp::SolidQueue] Failed to update heartbeats: #{e.class} - #{e.message}")
         raise if PatientHttp.testing?

data/lib/patient_http/solid_queue.rb

@@ -34,6 +34,11 @@ module PatientHttp
   module SolidQueue
     VERSION = File.read(File.join(__dir__, "../../VERSION")).strip
 
+    # Raised when the crash-recovery registry entry for a request cannot be
+    # written. The request is rejected rather than accepted without a durable
+    # record, and the job retries.
+    class RegistrationError < StandardError; end
+
     autoload :CallbackJob, File.join(__dir__, "solid_queue/callback_job")
     autoload :Configuration, File.join(__dir__, "solid_queue/configuration")
     autoload :Context, File.join(__dir__, "solid_queue/context")
@@ -49,13 +54,15 @@ module PatientHttp
     autoload :TaskMonitor, File.join(__dir__, "solid_queue/task_monitor")
     autoload :TaskMonitorThread, File.join(__dir__, "solid_queue/task_monitor_thread")
 
-    @processor = nil
+    @processors = {}
     @configuration = nil
     @after_completion_callbacks = []
     @after_error_callbacks = []
     @external_storage = nil
     @request_handler = nil
     @lifecycle_mutex = Mutex.new
+    @task_monitor = nil
+    @monitor_thread = nil
 
     class << self
       attr_writer :configuration
@@ -109,32 +116,32 @@ module PatientHttp
         @after_error_callbacks << block
       end
 
-      # Check if the processor is running.
+      # Check if any processor is running.
       #
       # @return [Boolean]
       def running?
-        !!@processor&.running?
+        @processors.values.any?(&:running?)
       end
 
-      # Check if the processor is draining (not accepting new requests).
+      # Check if any processor is draining (not accepting new requests).
       #
       # @return [Boolean]
       def draining?
-        !!@processor&.draining?
+        @processors.values.any?(&:draining?)
       end
 
-      # Check if the processor is stopping.
+      # Check if any processor is stopping.
       #
       # @return [Boolean]
       def stopping?
-        !!@processor&.stopping?
+        @processors.values.any?(&:stopping?)
       end
 
-      # Check if the processor is stopped.
+      # Check if all processors are stopped or none have been started.
       #
       # @return [Boolean]
       def stopped?
-        @processor.nil? || @processor.stopped?
+        @processors.values.all?(&:stopped?)
       end
 
       # Get an ExternalStorage instance for storing and fetching payloads.
@@ -152,12 +159,23 @@ module PatientHttp
       #   instance methods, or its fully qualified class name.
       # @param callback_args [#to_h, nil] Arguments to pass to callback
       # @param raise_error_responses [Boolean] If true, treats non-2xx responses as errors
+      # @param processor [Symbol, String, nil] Name of the processor profile that should
+      #   execute the request. Defaults to the request's own processor name or :default.
       # @return [String] the request ID
-      def execute(request, callback:, callback_args: nil, raise_error_responses: false)
+      # @raise [PatientHttp::UnknownProcessorError] if the processor profile is not configured
+      def execute(request, callback:, callback_args: nil, raise_error_responses: false, processor: nil)
         PatientHttp::CallbackValidator.validate!(callback)
         callback_name = callback.is_a?(Class) ? callback.name : callback.to_s
         callback_args = PatientHttp::CallbackValidator.validate_callback_args(callback_args)
         request_id = SecureRandom.uuid
+        processor_name = (processor || request.processor || :default).to_s
+
+        # Catch a misspelled profile name at the call site. A job that names an
+        # unconfigured profile is retried instead, which covers rolling deploys
+        # where the executing process is older than the enqueueing one.
+        unless configuration.processor_profiles.key?(processor_name.to_sym)
+          raise PatientHttp::UnknownProcessorError.new("No processor profile configured for #{processor_name.inspect}")
+        end
 
         encrypted = encrypt(request.as_json)
 
@@ -167,38 +185,59 @@ module PatientHttp
           encrypted
         end
 
-        RequestJob.perform_later(data, callback_name, raise_error_responses, callback_args, request_id)
+        RequestJob.perform_later(data, callback_name, raise_error_responses, callback_args, request_id, processor_name)
 
         request_id
       end
 
-      # Start the processor.
+      # Start a processor for each configured processor profile, along with
+      # the shared crash-recovery monitor.
       #
       # @return [void]
       def start
         @lifecycle_mutex.synchronize do
-          return if @processor && !@processor.stopped?
+          return if @processors.any? && !@processors.values.all?(&:stopped?)
 
-          @processor = PatientHttp::Processor.new(configuration)
-          @processor.observe(ProcessorObserver.new(@processor))
-          @processor.start
+          @task_monitor ||= TaskMonitor.new(
+            configuration,
+            max_connections: -> { @processors.values.sum { |p| p.config.max_connections } }
+          )
+
+          @processors = {}
+          configuration.processor_profiles.each_key do |name|
+            processor = PatientHttp::Processor.new(configuration.processor_config(name), name: name)
+            processor.observe(ProcessorObserver.new(processor, task_monitor: @task_monitor))
+            @processors[name] = processor
+          end
+          @processors.each_value(&:start)
+
+          # A previous run can leave a monitor thread behind if the processors
+          # stopped without going through #stop.
+          @monitor_thread&.stop
+
+          @monitor_thread = TaskMonitorThread.new(
+            configuration,
+            @task_monitor,
+            -> { @processors.values.flat_map(&:tracked_request_ids) }
+          )
+          @monitor_thread.start
         end
 
         register_handler
       end
 
-      # Signal the processor to drain (stop accepting new requests).
+      # Signal all processors to drain (stop accepting new requests).
       #
       # @return [void]
       def quiet
         @lifecycle_mutex.synchronize do
           return unless running?
 
-          @processor.drain
+          @processors.each_value(&:drain)
         end
       end
 
-      # Stop the processor gracefully.
+      # Stop all processors gracefully.
       #
       # @param timeout [Float, nil] maximum time to wait for in-flight requests to complete
       # @return [void]
@@ -208,10 +247,11 @@ module PatientHttp
         end
 
         @lifecycle_mutex.synchronize do
-          return unless @processor
+          return if @processors.empty?
 
-          @processor.stop(timeout: timeout)
-          @processor = nil
+          stop_processors(timeout: timeout)
+          @processors = {}
+          shutdown_shared_services
         end
       end
 
@@ -224,8 +264,9 @@ module PatientHttp
           PatientHttp.unregister_handler(@request_handler)
         end
         @lifecycle_mutex.synchronize do
-          @processor&.stop(timeout: 0)
-          @processor = nil
+          stop_processors(timeout: 0)
+          @processors = {}
+          shutdown_shared_services
         end
         @configuration = nil
         @external_storage = nil
@@ -292,11 +333,55 @@ module PatientHttp
         configuration.encryptor.decrypt(value)
       end
 
-      # Returns the processor instance.
+      # Returns a processor instance by name (internal accessor).
       #
+      # @param name [Symbol, String] the processor name
       # @return [PatientHttp::Processor, nil]
       # @api private
-      attr_accessor :processor
+      def processor(name = :default)
+        @processors[name.to_sym]
+      end
+
+      # Set the default processor (internal, for testing).
+      #
+      # @param value [PatientHttp::Processor, nil]
+      # @api private
+      def processor=(value)
+        if value.nil?
+          @processors.delete(:default)
+        else
+          @processors[:default] = value
+        end
+      end
+
+      private
+
+      # Stop every processor, draining them at the same time so the timeout
+      # bounds the whole shutdown instead of each processor in turn.
+      def stop_processors(timeout:)
+        processors = @processors.values
+        return if processors.empty?
+
+        if processors.one?
+          processors.first.stop(timeout: timeout)
+        else
+          processors.map { |processor| Thread.new { processor.stop(timeout: timeout) } }.each(&:join)
+        end
+      end
+
+      # Stop the shared monitor thread and remove this process from the
+      # registry. Called with the lifecycle mutex held after all processors
+      # have stopped.
+      def shutdown_shared_services
+        @monitor_thread&.stop
+        @monitor_thread = nil
+        begin
+          @task_monitor&.remove_process
+        rescue => e
+          configuration.logger&.error("[PatientHttp::SolidQueue] Failed to remove process registration: #{e.inspect}")
+        end
+        @task_monitor = nil
+      end
     end
   end
 end

data/patient_http-solid_queue.gemspec

@@ -37,8 +37,6 @@ Gem::Specification.new do |spec|
 
   spec.required_ruby_version = ">= 3.2"
 
-  spec.add_dependency "patient_http", ">= 1.3.0"
+  spec.add_dependency "patient_http", ">= 1.5.0"
   spec.add_dependency "solid_queue", ">= 1.0.0"
-
-  spec.add_development_dependency "bundler"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: patient_http-solid_queue
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Brian Durand
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.3.0
+        version: 1.5.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.3.0
+        version: 1.5.0
 - !ruby/object:Gem::Dependency
   name: solid_queue
   requirement: !ruby/object:Gem::Requirement
@@ -37,20 +37,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 1.0.0
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 email:
 - bbdurand@gmail.com
 executables: []