patient_http-solid_queue 1.0.1 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c4ac3bbb362f31671cf29f2ceb4f848c691a810fe0ee5a75cdb688a8e3496b7
-  data.tar.gz: 38743ad3ff5bf5a28d59b0b2de7f4d023d5d06deee198dcfc90d596682453976
+  metadata.gz: 4b17f1e1d5c477d8ca2dac603c0521dd7a842469d24db38dadb5f24858443dca
+  data.tar.gz: 25d38a530f6ec873e5b224462d0335fa99912e6c1c36cad46f2c072ff9a5162a
 SHA512:
-  metadata.gz: bbd0c898f04fca1cc69fb8e43d2af88f48003975a3d27cd91d89aacad9fb1da7fbbcb2299d51f81b8c04bbf892893982f4a63820ae47644f0492d165f231ae32
-  data.tar.gz: c36e0b07297193eebe1b42c25890f51d7b6d0df33304f2e57a53e8c94605c662c8db97cb3bd5641b66c40fea2cd32aac7f76d5eee38e3e77c58f21850da875be
+  metadata.gz: f1d0021667606b1b527eb4b1013fb418c5e9593d89d8daba3102415305f3a337d9258fd9c4387474c0f913b421619ceeeead1d59495fdf5d814635e1df2760b0
+  data.tar.gz: b27c9cfa86a5265061b3510bd168f7780b61bab702efc621d2214dac72c4bc79efb58c3c814c01a6ad71b7a7d986b85697d73e608ccd3bdfa8834a692f9f4555

data/CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.0.2
+
+### Fixed
+
+- Externally stored request payloads are no longer deleted as soon as the request is submitted to the processor. They are now retained until the request completes so that Active Job retries, shutdown re-enqueues, and crash recovery can still fetch them. Discarded `RequestJob` jobs clean up their stored payloads via an `after_discard` hook.
+- `RequestJob` now declares `retry_on` for `PatientHttp::MaxCapacityError` and `PatientHttp::NotRunningError` with a polynomial backoff. Previously these errors (raised as normal backpressure when the processor is at max capacity) sent jobs straight to the failed jobs list because Active Job does not retry by default.
+- `CallbackJob` no longer deletes externally stored payloads when the callback raises, so retries can fetch the payload and re-run the callback.
+- Crash recovery no longer silently loses a request if the process crashes between removing the inflight record and re-enqueueing the job. The record is now claimed first and only deleted after the job has been enqueued, so a failed recovery attempt is retried by a later garbage collection pass.
+- Fixed process registration and garbage collection on MySQL, where passing an explicit conflict target to `upsert`/`insert_all` raises an error. Crash recovery was silently disabled on MySQL.
+- `PatientHttp::SolidQueue.configure` and `reset_configuration!` now reset the memoized external storage so it picks up the new configuration.
+- Fixed race conditions that could create duplicate processors or task monitor threads from concurrent lifecycle calls; starting while the processor is draining no longer replaces it.
+- The task monitor thread no longer holds a database connection from the pool while sleeping between heartbeats.
+
 ## 1.0.1
 
 ### Fixed

data/README.md

@@ -94,6 +94,14 @@ The `response.callback_args` and `error.callback_args` provide access to the arg
 
 > [!IMPORTANT]
 > Do not re-raise errors in the `on_error` callback as a means to retry the request. That will just retry the error callback job. If you want to retry the original request, you can enqueue a new request from within `on_error`. Be careful with this approach, though, as it can lead to infinite retry loops if the error condition is not resolved.
+
+Callback jobs are not retried by default. If you want failed callbacks to be retried before being discarded, configure `retry_on` in an initializer:
+
+```ruby
+PatientHttp::SolidQueue::CallbackJob.retry_on StandardError, wait: :polynomially_longer, attempts: 5
+```
+
+Note that Active Job runs `after_discard` hooks for any unhandled exception, not just when configured retries are exhausted. Without `retry_on`, the first callback failure will trigger the `on_retries_exhausted` handler and delete any externally stored payload, even though the failed job can still be retried manually from Mission Control (such retries will fail if the payload was stored externally).
 >
 > Also note that the error callback is only called when an exception occurs during the HTTP request (timeout, connection failure, etc). HTTP error status codes (4xx, 5xx) do not trigger the error callback by default. Instead, they are treated as completed requests and passed to the `on_complete` callback. See the "Handling HTTP Error Responses" section below for how to treat HTTP errors as exceptions.
 
@@ -436,7 +444,7 @@ See the [Configuration](lib/patient_http/solid_queue/configuration.rb) class for
 
 > [!IMPORTANT]
 >
-> One difference between using this gem and making synchronous HTTP requests from a Solid Queue job is that if `max_connections` is reached due to slow asynchronous requests, new requests will trigger an error on the Active Job. The Active Job retry mechanism will handle re-enqueuing the job.
+> One difference between using this gem and making synchronous HTTP requests from a Solid Queue job is that if `max_connections` is reached due to slow asynchronous requests, new requests will trigger an error on the Active Job. The job declares `retry_on` for this error with a polynomial backoff, so it will automatically be retried until the processor has capacity again.
 >
 > In contrast, slow synchronous HTTP requests will fill up the worker pool and block new jobs from being dequeued until a worker thread becomes free.
 >

data/VERSION

@@ -1 +1 @@
-1.0.1
+1.0.2

data/lib/patient_http/solid_queue/callback_job.rb

@@ -52,21 +52,22 @@ module PatientHttp
         actual_data = ref_data ? PatientHttp::SolidQueue.external_storage.fetch(data) : data
         actual_data = PatientHttp::SolidQueue.decrypt(actual_data)
 
-        begin
-          if result_type == "response"
-            response = PatientHttp::Response.load(actual_data)
-            PatientHttp::SolidQueue.invoke_completion_callbacks(response)
-            callback_service.on_complete(response)
-          elsif result_type == "error"
-            error = PatientHttp::Error.load(actual_data)
-            PatientHttp::SolidQueue.invoke_error_callbacks(error)
-            callback_service.on_error(error)
-          else
-            raise ArgumentError, "Unknown result_type: #{result_type}"
-          end
-        ensure
-          PatientHttp::SolidQueue.external_storage.delete(ref_data) if ref_data
+        if result_type == "response"
+          response = PatientHttp::Response.load(actual_data)
+          PatientHttp::SolidQueue.invoke_completion_callbacks(response)
+          callback_service.on_complete(response)
+        elsif result_type == "error"
+          error = PatientHttp::Error.load(actual_data)
+          PatientHttp::SolidQueue.invoke_error_callbacks(error)
+          callback_service.on_error(error)
+        else
+          raise ArgumentError, "Unknown result_type: #{result_type}"
         end
+
+        # Only delete the stored payload after the callback succeeds so that
+        # retries can still fetch it. Discarded jobs are cleaned up by the
+        # after_discard hook.
+        PatientHttp::SolidQueue.external_storage.delete(ref_data) if ref_data
       end
     end
   end

data/lib/patient_http/solid_queue/request_job.rb

@@ -10,36 +10,51 @@ module PatientHttp
     #
     # @api private
     class RequestJob < ActiveJob::Base
+      # Rejection due to backpressure is part of normal operation: retry until the
+      # processor has capacity again. NotRunningError covers jobs that run during
+      # the narrow window when the processor is draining or stopping.
+      retry_on PatientHttp::MaxCapacityError, PatientHttp::NotRunningError, wait: :polynomially_longer, attempts: :unlimited
+
       # Capture the Active Job serialized hash into Context so RequestExecutor can use it.
       around_perform do |job, block|
         PatientHttp::SolidQueue::Context.with_job(job.serialize) { block.call }
       end
 
+      # Clean up the externally stored request payload when the job is discarded.
+      # The payload is normally deleted by TaskHandler when the request completes,
+      # so this only fires for requests that never made it that far.
+      after_discard do |job, _exception|
+        PatientHttp::SolidQueue.external_storage.delete(job.arguments[0])
+      rescue => e
+        PatientHttp::SolidQueue.configuration.logger&.warn(
+          "[PatientHttp::SolidQueue] Failed to delete stored payload for dead job: #{e.class.name} #{e.message}".strip
+        )
+      end
+
       # @param data [Hash] Request data (possibly a storage reference)
       # @param callback_service_name [String] Fully qualified callback service class name
       # @param raise_error_responses [Boolean, nil] Whether to treat non-2xx responses as errors
       # @param callback_args [Hash, nil] Arguments to pass to the callback
       # @param request_id [String, nil] Unique request ID for tracking
       def perform(data, callback_service_name, raise_error_responses, callback_args, request_id)
-        ref_data = PatientHttp::ExternalStorage.storage_ref?(data) ? data : nil
-        actual_data = ref_data ? PatientHttp::SolidQueue.external_storage.fetch(data) : data
+        actual_data = PatientHttp::ExternalStorage.storage_ref?(data) ? PatientHttp::SolidQueue.external_storage.fetch(data) : data
         actual_data = PatientHttp::SolidQueue.decrypt(actual_data)
 
         request = PatientHttp::Request.load(actual_data)
         active_job_data = PatientHttp::SolidQueue::Context.current_job
 
-        begin
-          RequestExecutor.execute(
-            request,
-            callback: callback_service_name,
-            raise_error_responses: raise_error_responses,
-            callback_args: callback_args,
-            active_job_data: active_job_data,
-            request_id: request_id
-          )
-        ensure
-          PatientHttp::SolidQueue.external_storage.delete(ref_data) if ref_data
-        end
+        # The stored payload must not be deleted here: this job data is re-enqueued
+        # for Active Job retries (e.g. MaxCapacityError), processor shutdown retries,
+        # and crash recovery, all of which need to fetch the payload again.
+        # TaskHandler deletes it when the request completes.
+        RequestExecutor.execute(
+          request,
+          callback: callback_service_name,
+          raise_error_responses: raise_error_responses,
+          callback_args: callback_args,
+          active_job_data: active_job_data,
+          request_id: request_id
+        )
       end
     end
   end

data/lib/patient_http/solid_queue/task_handler.rb

@@ -18,11 +18,13 @@ module PatientHttp
       def on_complete(response, callback)
         data = store_if_needed(response.as_json)
         CallbackJob.perform_later(data, "response", callback)
+        delete_stored_request_payload
       end
 
       def on_error(error, callback)
         data = store_if_needed(error.as_json)
         CallbackJob.perform_later(data, "error", callback)
+        delete_stored_request_payload
       end
 
       def retry
@@ -39,6 +41,24 @@ module PatientHttp
 
       private
 
+      # Delete the externally stored request payload once the request has
+      # completed. Until then the payload must remain fetchable because the
+      # Active Job data referencing it can be re-enqueued by Active Job retries,
+      # processor shutdown retries, and crash recovery. Only applies to
+      # RequestJob jobs; other job types own their own arguments.
+      def delete_stored_request_payload
+        return unless @active_job_data["job_class"] == RequestJob.name
+
+        data = @active_job_data["arguments"]&.first
+        return unless PatientHttp::ExternalStorage.storage_ref?(data)
+
+        PatientHttp::SolidQueue.external_storage.delete(data)
+      rescue => e
+        PatientHttp::SolidQueue.configuration.logger&.warn(
+          "[PatientHttp::SolidQueue] Failed to delete stored request payload: #{e.class.name} #{e.message}".strip
+        )
+      end
+
       def store_if_needed(data)
         encrypted = PatientHttp::SolidQueue.encrypt(data)
         external_storage = PatientHttp::SolidQueue.external_storage

data/lib/patient_http/solid_queue/task_monitor.rb

@@ -79,7 +79,7 @@ module PatientHttp
       def ping_process
         ProcessRegistration.upsert(
           {process_id: @lock_identifier, max_connections: @config.max_connections, last_seen_at: Time.current},
-          unique_by: :process_id
+          unique_by: upsert_unique_by(:process_id)
         )
       rescue => e
         @config.logger&.error("[PatientHttp::SolidQueue] Failed to ping process: #{e.message}")
@@ -211,31 +211,47 @@ module PatientHttp
       private
 
       def ensure_gc_lock_row!
-        GcLock.insert_all([{lock_name: GC_LOCK_NAME}], unique_by: :lock_name)
+        GcLock.insert_all([{lock_name: GC_LOCK_NAME}])
       end
 
-      # Re-enqueue a single orphaned record atomically.
+      # MySQL does not support an explicit conflict target for upserts; it always
+      # resolves conflicts via the table's unique indexes (ON DUPLICATE KEY UPDATE).
+      # Adapters with conflict targets (PostgreSQL, SQLite) require one to be given.
       #
-      # Uses a delete-by-exact-heartbeat to handle race conditions: if the
-      # heartbeat was updated between our read and the delete, the delete
-      # returns 0 rows and we skip re-enqueueing.
+      # @param column [Symbol] the unique column to use as the conflict target
+      # @return [Symbol, nil] the column, or nil when the adapter forbids a target
+      def upsert_unique_by(column)
+        Record.connection.supports_insert_conflict_target? ? column : nil
+      end
+
+      # Re-enqueue a single orphaned record.
+      #
+      # Uses a claim-by-exact-heartbeat update to handle race conditions: if the
+      # heartbeat was updated between our read and the claim, the update returns
+      # 0 rows and we skip re-enqueueing. Claiming (refreshing the heartbeat)
+      # before enqueueing means a crash mid-recovery leaves the record behind to
+      # go stale and be retried by a later GC pass, instead of losing the request.
+      # The record is only deleted after the job has been enqueued.
       #
       # @param record [InflightRequest] the orphaned record
       # @param threshold [Float] heartbeat threshold (only records below this are orphaned)
       # @param logger [Logger] logger for output
       # @return [Boolean] true if successfully re-enqueued
       def reenqueue_orphaned_record(record, threshold, logger)
-        # Atomically remove only if still orphaned (heartbeat unchanged)
-        deleted = InflightRequest
+        # Atomically claim only if still orphaned (heartbeat unchanged). The dead
+        # process_id is left in place so a failed recovery becomes orphaned again.
+        claimed = InflightRequest
           .where(task_id: record.task_id, heartbeat_at: record.heartbeat_at)
           .where("heartbeat_at < ?", threshold)
-          .delete_all
+          .update_all(heartbeat_at: Time.current)
 
-        return false if deleted == 0
+        return false if claimed == 0
 
         job_data = JSON.parse(record.job_payload)
         ActiveJob::Base.deserialize(job_data).tap { |j| j.executions = 0 }.enqueue
 
+        InflightRequest.where(task_id: record.task_id).delete_all
+
         logger&.info(
           "[PatientHttp::SolidQueue] Re-enqueued orphaned request #{record.task_id} to #{job_data["job_class"]}"
         )

data/lib/patient_http/solid_queue/task_monitor_thread.rb

@@ -34,11 +34,10 @@ module PatientHttp
       #
       # @return [void]
       def start
-        return if @running.true?
-        @running.make_true
+        return unless @running.make_true
         @stop_signal.reset
 
-        @task_monitor.ping_process
+        with_connection { @task_monitor.ping_process }
 
         @thread = Thread.new do
           run
@@ -80,16 +79,24 @@ module PatientHttp
           break unless @running.true?
 
           current_time = monotonic_time
-
-          if current_time - last_heartbeat_update >= @config.heartbeat_interval
-            @task_monitor.ping_process
-            update_heartbeats
-            last_heartbeat_update = current_time
-          end
-
-          if current_time - last_gc_attempt >= @config.heartbeat_interval
-            attempt_garbage_collection
-            last_gc_attempt = current_time
+          update_heartbeats_due = current_time - last_heartbeat_update >= @config.heartbeat_interval
+          gc_attempt_due = current_time - last_gc_attempt >= @config.heartbeat_interval
+
+          if update_heartbeats_due || gc_attempt_due
+            # Check out a database connection only for the duration of the work
+            # so the thread does not hold a slot from the pool while it sleeps.
+            with_connection do
+              if update_heartbeats_due
+                @task_monitor.ping_process
+                update_heartbeats
+                last_heartbeat_update = current_time
+              end
+
+              if gc_attempt_due
+                attempt_garbage_collection
+                last_gc_attempt = current_time
+              end
+            end
           end
 
           wait_time = @config.heartbeat_interval / 2.0
@@ -100,6 +107,10 @@ module PatientHttp
         @config.logger&.info("[PatientHttp::SolidQueue] Monitor thread stopped")
       end
 
+      def with_connection(&block)
+        Record.connection_pool.with_connection(&block)
+      end
+
       def update_heartbeats
         request_ids = @inflight_ids_callback.call
         return if request_ids.empty?

data/lib/patient_http/solid_queue.rb

@@ -55,6 +55,7 @@ module PatientHttp
     @after_error_callbacks = []
     @external_storage = nil
     @request_handler = nil
+    @lifecycle_mutex = Mutex.new
 
     class << self
       attr_writer :configuration
@@ -66,8 +67,10 @@ module PatientHttp
       def configure
         configuration = Configuration.new
         yield(configuration) if block_given?
-        register_handler
         @configuration = configuration
+        @external_storage = nil
+        register_handler
+        configuration
       end
 
       # Return the current configuration, initializing with defaults if necessary.
@@ -82,6 +85,7 @@ module PatientHttp
       # @return [Configuration]
       def reset_configuration!
         @configuration = nil
+        @external_storage = nil
         configuration
       end
 
@@ -168,11 +172,13 @@ module PatientHttp
       #
       # @return [void]
       def start
-        return if running?
+        @lifecycle_mutex.synchronize do
+          return if @processor && !@processor.stopped?
 
-        @processor = PatientHttp::Processor.new(configuration)
-        @processor.observe(ProcessorObserver.new(@processor))
-        @processor.start
+          @processor = PatientHttp::Processor.new(configuration)
+          @processor.observe(ProcessorObserver.new(@processor))
+          @processor.start
+        end
 
         register_handler
       end
@@ -181,9 +187,11 @@ module PatientHttp
       #
       # @return [void]
       def quiet
-        return unless running?
+        @lifecycle_mutex.synchronize do
+          return unless running?
 
-        @processor.drain
+          @processor.drain
+        end
       end
 
       # Stop the processor gracefully.
@@ -191,14 +199,16 @@ module PatientHttp
       # @param timeout [Float, nil] maximum time to wait for in-flight requests to complete
       # @return [void]
       def stop(timeout: nil)
-        return unless @processor
-
         if @request_handler
           PatientHttp.unregister_handler(@request_handler)
         end
 
-        @processor.stop(timeout: timeout)
-        @processor = nil
+        @lifecycle_mutex.synchronize do
+          return unless @processor
+
+          @processor.stop(timeout: timeout)
+          @processor = nil
+        end
       end
 
       # Reset all state (useful for testing).
@@ -209,8 +219,10 @@ module PatientHttp
         if @request_handler
           PatientHttp.unregister_handler(@request_handler)
         end
-        @processor&.stop(timeout: 0)
-        @processor = nil
+        @lifecycle_mutex.synchronize do
+          @processor&.stop(timeout: 0)
+          @processor = nil
+        end
         @configuration = nil
         @external_storage = nil
         @after_completion_callbacks = []

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: patient_http-solid_queue
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.0.2
 platform: ruby
 authors:
 - Brian Durand