patient_http 1.3.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ba7400987a1a1483e88f8c0e3c3fb3579544a02f5f3bdb6da7752f5864792fcb
-  data.tar.gz: cc1ab11246151f2b9843e81c730653e922a34db6f4a57600267637fbe72d7f49
+  metadata.gz: d0888d7e92f43f2031bf7f0eb5b0f4dd528ac7b79957b506f1df61825309990a
+  data.tar.gz: 87b5817b8205a0b605edb7cb41facf0f6b5690122ff63cb669dbcc6c6b090cd0
 SHA512:
-  metadata.gz: e021b76930cc0c1d6c0741723d2db084947f04b8daee1564bc1b04d56d24399443dc6567fe153a99795d5c322810ab56f9cd218f1b8ddafb3a127785352f0bb7
-  data.tar.gz: 5d920b85e033e9fbfcd551ca2f7702d260da8ddbea04f105173e591c07fab7b66df2c6d6a85303463fa6faa6d2c8e6357296c317e6ccec8e7599dbbd97b1f842
+  metadata.gz: 4bd3586a382436ac11872d10380afb28326497c9538383ae0e884ecaf901dd952df0202e9ed740047b6336c60960ebff20b83abcdc86a585c67e0915097dbe92
+  data.tar.gz: d86c67c48c1aa03d0aa47f4675626696ab9012b526add67a4d8ebf611aca40b8229e722bd2e07c2ef6de6caa6110a01507d3317f029668fa89a768d145fea299

data/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.4.0
+
+### Added
+
+ - `ProcessorObserver` events for the full task pipeline: `request_enqueued` is sent when a task is announced to the processor, before the task is visible to the reactor, so observers can set up durable tracking (e.g. a crash-recovery registry entry) before `Processor#enqueue` returns or raises. `request_rejected` is sent when an announced task is not accepted (not running or at capacity). `request_requeued` is sent when an incomplete task is re-enqueued through its task handler, so observers can tear down tracking for tasks the job system owns again. Redirect tasks are announced with `request_enqueued` as well.
+ - `Processor#tracked_request_ids` returns the IDs of all tasks in the pipeline (queued, pending, and in-flight), so durable tracking can keep heartbeats alive for tasks that have not started yet.
+
+### Fixed
+
+- A task is now marked as started before the `request_start` notification is sent. Before this fix, a shutdown that snapshotted the task between the two steps re-enqueued the task without a `request_end` notification, which could leak observer tracking for the task and cause a duplicate execution through crash recovery.
+
 ## 1.3.0
 
 ### Added

data/README.md

@@ -744,6 +744,14 @@ end
 processor.observe(MetricsObserver.new)
 ```
 
+Observers can also track the full task pipeline:
+
+ - `request_enqueued(request_task)` is called when a task is announced to the processor (before the task is visible to the reactor). It is guaranteed to arrive before `request_start`, so observers can set up durable tracking (e.g. a crash-recovery registry entry) before `Processor#enqueue` returns or raises.
+ - `request_rejected(request_task)` is called when an announced task is not accepted (not running or at capacity), so observers can tear down anything they set up in `request_enqueued`.
+- `request_requeued(request_task)` is called when an incomplete task is re-enqueued through its task handler (processor shutdown or reactor failure). The job system owns the request again once this is sent.
+
+Use `Processor#tracked_request_ids` to get the IDs of all tasks in the pipeline (queued, pending, and in-flight), for example to keep heartbeats alive for tasks that have not started yet.
+
 ## Testing
 
 Use `SynchronousExecutor` to execute requests synchronously in tests. This class can be used in place of the async processor for testing your request handling logic without needing to start the full async infrastructure.

data/VERSION

@@ -1 +1 @@
-1.3.0
+1.4.0

data/lib/patient_http/processor.rb

@@ -33,6 +33,10 @@ module PatientHttp
       @reactor_generation = 0
       @inflight_requests = Concurrent::Hash.new
       @pending_tasks = Concurrent::Hash.new
+      # Tasks pushed onto @queue but not yet popped by the reactor. Kept in a
+      # hash because Thread::Queue cannot be enumerated; used to report all
+      # tracked task ids (e.g. for heartbeat updates on queued tasks).
+      @queued_tasks = Concurrent::Hash.new
       @tasks_lock = Mutex.new
       @idle_condition = ConditionVariable.new
       @testing_callback = nil
@@ -76,14 +80,21 @@ module PatientHttp
           # lock; re-enqueueing runs outside it. This is idempotent with stop()'s
           # reenqueue_pending_requests: whichever runs second snapshots an empty
           # set.
+          current_generation = false
           orphaned_tasks = @tasks_lock.synchronize do
             if @reactor_generation == generation
+              current_generation = true
               drain_tracked_tasks_locked
             else
               []
             end
           end
           reenqueue_tasks(orphaned_tasks)
+          # Hand back tasks still sitting in the queue as well; a reactor that
+          # exits without a stop() call is the last owner of those tasks.
+          # stop() performs the same drain after reaping the reactor, and
+          # whichever drain runs second finds nothing left.
+          reenqueue_remaining_queue_items if current_generation
         end
 
         # The transition can fail if the reactor thread already failed and
@@ -198,22 +209,18 @@ module PatientHttp
     # @raise [MaxCapacityError] if at max capacity
     # @return [void]
     def enqueue(task)
-      at_capacity = false
+      raise NotRunningError.new("Cannot enqueue request: processor is #{state}") unless running?
 
-      @tasks_lock.synchronize do
+      accepted = announce_and_enqueue(task) do
+        # The pre-check above is advisory; re-check the running state under
+        # the lock since observers were notified outside of it.
         raise NotRunningError.new("Cannot enqueue request: processor is #{state}") unless running?
 
-        # Check capacity - raise error if at max connections
-        total = @queue.size + @pending_tasks.size + @inflight_requests.size
-        if total >= @config.max_connections
-          at_capacity = true
-        else
-          task.enqueued!
-          @queue.push(task)
-        end
+        # Check capacity - the task is only accepted below max connections.
+        @queue.size + @pending_tasks.size + @inflight_requests.size < @config.max_connections
       end
 
-      if at_capacity
+      unless accepted
         notify_observers { |observer| observer.capacity_exceeded }
         raise MaxCapacityError.new("Cannot enqueue request: already at max capacity (#{@config.max_connections} connections)")
       end
@@ -307,6 +314,17 @@ module PatientHttp
       end
     end
 
+    # Get the IDs of all tasks in the pipeline (queued, pending, and in-flight).
+    # Use this to keep durable tracking (e.g. heartbeats) alive for tasks the
+    # processor has accepted but not yet started.
+    #
+    # @return [Array<String>]
+    def tracked_request_ids
+      @tasks_lock.synchronize do
+        (@queued_tasks.keys + @pending_tasks.keys + @inflight_requests.keys).uniq
+      end
+    end
+
     # Add an observer for processor events.
     #
     # @param observer [ProcessorObserver] the observer to add
@@ -393,6 +411,7 @@ module PatientHttp
 
           # Track as pending immediately to avoid race condition with stop()
           @tasks_lock.synchronize do
+            @queued_tasks.delete(request_task.id)
             @pending_tasks[request_task.id] = request_task
           end
 
@@ -473,12 +492,15 @@ module PatientHttp
 
         @pending_tasks.delete(task.id)
         @inflight_requests[task.id] = task
+        # Mark the task started in the same locked section that tracks it so
+        # a shutdown snapshot always sees a consistent started state. The
+        # shutdown re-enqueue path uses started? to pair request_end with
+        # request_start.
+        task.started!
       end
 
       notify_observers { |observer| observer.request_start(task) }
 
-      # Mark task as started
-      task.started!
       claimed = false
 
       begin
@@ -611,13 +633,9 @@ module PatientHttp
       # from in-flight) and enqueue the redirect. If the claim fails the
       # shutdown sequence already re-enqueued the original, so drop the redirect.
       redirect_task = task.redirect_task(location: location, status: status)
-      redirect_task.enqueued!
-
-      claimed = @tasks_lock.synchronize do
-        next false if @inflight_requests.delete(task.id).nil?
 
-        @queue.push(redirect_task)
-        true
+      claimed = announce_and_enqueue(redirect_task) do
+        !@inflight_requests.delete(task.id).nil?
       end
       return false unless claimed
 
@@ -646,6 +664,47 @@ module PatientHttp
       raise if PatientHttp.testing?
     end
 
+    # Announce a task to observers and make it visible to the reactor. The
+    # task is announced before it can start, finish, or be re-enqueued, so
+    # observers can set up durable tracking first. Errors from the
+    # request_enqueued announcement propagate and reject the task, because a
+    # failed tracking setup must not let the task be accepted as if it were
+    # durable. The block runs while {@tasks_lock} is held and decides whether
+    # the task is accepted; when it returns false or raises, observers receive
+    # request_rejected so they can tear down anything they set up for the
+    # request_enqueued announcement. The rejection notification never replaces
+    # an exception that is already being raised.
+    #
+    # @param task [RequestTask] the request task to announce and enqueue
+    # @return [Boolean] true if the task was accepted
+    def announce_and_enqueue(task)
+      task.enqueued!
+      accepted = false
+
+      begin
+        notify_observers! { |observer| observer.request_enqueued(task) }
+
+        @tasks_lock.synchronize do
+          if yield
+            @queued_tasks[task.id] = task
+            @queue.push(task)
+            accepted = true
+          end
+        end
+      ensure
+        unless accepted
+          pending_error = $!
+          begin
+            notify_observers { |observer| observer.request_rejected(task) }
+          rescue
+            raise unless pending_error
+          end
+        end
+      end
+
+      accepted
+    end
+
     # Notify all observers of an event. Observers are called outside of any
     # internal lock so they can safely call back into the processor.
     def notify_observers(&block)
@@ -655,6 +714,16 @@ module PatientHttp
       end
     end
 
+    # Notify all observers of an event and let observer errors propagate.
+    # Used for notifications the caller must be able to react to, such as
+    # durable tracking setup in request_enqueued.
+    def notify_observers!
+      observers = @tasks_lock.synchronize { @observers.dup }
+      observers.each do |observer|
+        yield(observer)
+      end
+    end
+
     def notify_observer(observer)
       yield(observer)
     rescue => e
@@ -697,16 +766,27 @@ module PatientHttp
     end
 
     def reenqueue_remaining_queue_items
-      tasks_to_reenqueue = []
-
-      # Drain remaining items from the queue (skip nil sentinels from stop)
-      until @queue.empty?
-        begin
-          task = @queue.pop(true)
-          tasks_to_reenqueue << task if task
-        rescue ThreadError
-          break
+      tasks_to_reenqueue = @tasks_lock.synchronize do
+        tasks = []
+
+        # Drain remaining items from the queue (skip nil sentinels from stop)
+        until @queue.empty?
+          begin
+            task = @queue.pop(true)
+            tasks << task if task
+          rescue ThreadError
+            break
+          end
         end
+
+        tasks.each { |task| @queued_tasks.delete(task.id) }
+        # The reactor has exited and no new tasks can be accepted, so any id
+        # still tracked as queued belongs to a task that left the queue
+        # without reaching pending or in-flight tracking. Reclaim those tasks
+        # as well so they are not tracked forever.
+        tasks.concat(@queued_tasks.values)
+        @queued_tasks.clear
+        tasks
       end
 
       reenqueue_tasks(tasks_to_reenqueue)
@@ -715,6 +795,10 @@ module PatientHttp
     def reenqueue_tasks(tasks_to_reenqueue)
       tasks_to_reenqueue.each do |task|
         task.retry
+        # The task handler's job system owns the request again; let observers
+        # tear down any durable tracking for the task. Only sent after a
+        # successful retry so a failed retry leaves the tracking in place.
+        notify_observers { |observer| observer.request_requeued(task) }
         # Only emit request_end for tasks that actually started, so observers
         # that pair request_start/request_end (e.g. an in-flight gauge) stay
         # balanced. Queued-but-never-started tasks emit neither.

data/lib/patient_http/processor_observer.rb

@@ -24,6 +24,41 @@ module PatientHttp
     def capacity_exceeded
     end
 
+    # Called when a request task is handed to the processor, before the task is
+    # visible to the reactor. The notification is guaranteed to arrive before
+    # request_start for the task, so observers can set up durable tracking
+    # (e.g. a crash-recovery registry entry) with no risk that the task
+    # completes first. If the processor does not accept the task,
+    # request_rejected is sent afterward. Unlike other notifications, an error
+    # raised here propagates from Processor#enqueue and rejects the task, so a
+    # failed tracking setup does not let the task be accepted as if it were
+    # durable.
+    #
+    # @param request_task [RequestTask] the request task that was enqueued
+    # @return [void]
+    def request_enqueued(request_task)
+    end
+
+    # Called when a request task announced with request_enqueued was not
+    # accepted by the processor (not running or at capacity). Observers should
+    # tear down anything they set up in request_enqueued; the caller owns the
+    # request again once this is sent.
+    #
+    # @param request_task [RequestTask] the request task that was rejected
+    # @return [void]
+    def request_rejected(request_task)
+    end
+
+    # Called when an incomplete request task was re-enqueued through its task
+    # handler (processor shutdown or reactor failure). The task handler's job
+    # system owns the request again once this is sent, so observers should
+    # tear down any durable tracking for the task.
+    #
+    # @param request_task [RequestTask] the request task that was re-enqueued
+    # @return [void]
+    def request_requeued(request_task)
+    end
+
     # Called when a request starts processing.
     #
     # @param request_task [RequestTask] the request task that started

data/patient_http.gemspec

@@ -43,6 +43,4 @@ Gem::Specification.new do |spec|
   spec.add_dependency "async-http", "~> 0.60"
   spec.add_dependency "concurrent-ruby", "~> 1.2"
   spec.add_dependency "logger"
-
-  spec.add_development_dependency "bundler"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: patient_http
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.4.0
 platform: ruby
 authors:
 - Brian Durand
@@ -65,20 +65,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 description: This gem provides a dedicated async HTTP processor that uses Ruby's Fiber
   scheduler for non-blocking I/O. Application threads hand off HTTP requests to the
   processor and return immediately. The processor handles hundreds of concurrent HTTP
@@ -159,7 +145,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.3
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Generic async HTTP connection pool for Ruby applications using Fiber-based
   concurrency