patient_http-sidekiq 1.2.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e5ae5287613de3682de817fb6c434e6a1a33c21c1b8aa1dde7b7808c1733b927
-  data.tar.gz: 350e8f054d1047fe70482d9c3fc2f5473b077e1e7f1c64cc6f023c8bf76f62b4
+  metadata.gz: 0f463cc156486ac04ea624baf032e3ebd37d219149a720fd5044a864a9abf201
+  data.tar.gz: 0b294c6cb0df108ca53fd872c71d284fd82ab0a42284769eb2af7f04ca25bd76
 SHA512:
-  metadata.gz: d843ce929d22f48c20bc7f487b35d11421166218081f4fea2490440781d0a00839266c3cb1dec039611a6728e2213c3a71f0a966454ba40c1efc6ab466648ca7
-  data.tar.gz: 006cf97e99a6d27847b038c19065a88b09e28a94c7392fbee5af4253e9483652e8022b21e42c271840cfc8bff326fcfd490b137d1b08c8984760c68f6f7bc3df
+  metadata.gz: b9ab7cebbc773fb92baba7d58212a8fd3733392b4a93d995bdfa7ad3b03d70e1cd3637d8ebddc68c7178ca400b1adc1219ab65ef8cd3c7b9e4b852a0245b11e3
+  data.tar.gz: 994e51c5f8d7df504f6be4ee32bdd1e32607e62672bf7f0d228d9758af8f2e1b473844063c20951726b4d2ccc37f5eae7549ce4a3796633004195a82af1bcc23

data/ARCHITECTURE.md

@@ -112,14 +112,21 @@ sequenceDiagram
     participant Callback as Callback Service
 
     App->>Module: get(url, callback: MyCallback)
-    Module->>Sidekiq: Enqueue RequestWorker
-    Sidekiq->>ReqWorker: Execute job
-    ReqWorker->>Processor: submit(request, handler)
-    activate Processor
-    Note over Processor: Request queued<br/>in memory
-    Processor-->>ReqWorker: Returns immediately
-    ReqWorker-->>Sidekiq: Job completes
-    deactivate Processor
+
+    alt Processor running in this process (direct execution)
+        Module->>Processor: submit(request, handler)
+        Note over Module: DirectTaskHandler keeps the<br/>RequestWorker args for re-enqueue
+        Processor-->>Module: Returns immediately
+    else Processor not in this process
+        Module->>Sidekiq: Enqueue RequestWorker
+        Sidekiq->>ReqWorker: Execute job
+        ReqWorker->>Processor: submit(request, handler)
+        activate Processor
+        Note over Processor: Request queued<br/>in memory
+        Processor-->>ReqWorker: Returns immediately
+        ReqWorker-->>Sidekiq: Job completes
+        deactivate Processor
+    end
 
     Note over Sidekiq: Worker thread free<br/>to process other jobs
 
@@ -149,6 +156,8 @@ Key integration points:
 3. **CallbackWorker** invokes the user's callback service methods
 4. **ExternalStorage** handles large payloads transparently at each step
 
+Direct execution (enabled by default with `config.direct_execution`) skips the `RequestWorker` enqueue when the processor runs in the current process. The request gets a `DirectTaskHandler` that holds the `RequestWorker` job arguments, so every re-enqueue path (processor shutdown, crash recovery, and the at-capacity fallback) can enqueue the request as a normal `RequestWorker` job. The handler exposes a minimal job record for the crash-recovery registry because the orphan sweep pushes the stored record from another process. Requests made in a `with_sidekiq_options` block and requests made while `Sidekiq::Testing` is enabled always go through the queue, so that Sidekiq applies the options. Options set with `config.sidekiq_options` (including a `queue`) do not apply to direct-executed requests, because no Sidekiq job is created; set `config.direct_execution = false` to route every request through the configured queue. A failure to write the crash-recovery registry entry rejects the request and raises to the caller, the same as a failed enqueue.
+
 ## Component Relationships
 
 ```mermaid
@@ -377,13 +386,15 @@ In-flight requests are tracked in Redis to enable recovery when Sidekiq processe
 - Re-enqueues orphaned requests via `Sidekiq::Client.push`
 
 ### Recovery Process
-1. `ProcessorObserver` notifies `TaskMonitor` when requests start/complete
-2. `TaskMonitorThread` updates heartbeat timestamps in Redis
+1. `ProcessorObserver` registers a request with `TaskMonitor` when the processor accepts it (before `Processor#enqueue` returns) and unregisters it when the request completes or when a Sidekiq job owns the request again (rejected or re-enqueued)
+2. `TaskMonitorThread` updates heartbeat timestamps in Redis for all tracked requests (queued, pending, and in-flight)
 3. If a process crashes, heartbeat updates stop
 4. Other processes' monitor threads detect stale timestamps
 5. Orphaned requests are atomically removed and re-enqueued
 6. Prevents lost work during deployments or crashes
 
+Recovery gives at-least-once delivery. A crash between a re-enqueue and the removal of the registry entry can execute a request more than once, so callbacks must be idempotent. A request is durable once the submitting call returns; a crash during the call behaves like a failed enqueue.
+
 **Redis Keys:**
 - `sidekiq:patient_http:inflight_index` - Sorted set of request IDs by timestamp
 - `sidekiq:patient_http:inflight_jobs` - Hash of request payloads
@@ -401,6 +412,9 @@ PatientHttp::Sidekiq.configure do |config|
   # Sidekiq worker options (applied to both RequestWorker and CallbackWorker)
   config.sidekiq_options = {queue: "patient_http", retry: 5}
 
+  # Skip the Sidekiq queue when the processor runs in the current process
+  config.direct_execution = true
+
   # Encryption (for sensitive data in Sidekiq jobs; inherited from PatientHttp::Configuration)
   config.encryption_key = ENV["PATIENT_HTTP_ENCRYPTION_KEY"]
 
@@ -465,6 +479,13 @@ Fiber reactor processes request
 Response/Error received
 ```
 
+With direct execution (the default), a request made in a process with a running
+processor skips the enqueue and the `RequestWorker#perform` steps. A
+`DirectTaskHandler` holds the `RequestWorker` job arguments, and the request
+goes straight to the processor. The rest of the flow is identical, and the
+re-enqueue paths use the handler to enqueue a normal `RequestWorker` job when
+needed.
+
 ### Processing a Response
 
 ```

data/CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 1.4.0
+
+### Added
+
+- Dedicated Redis connection pool for the gem's own threads (`redis_pool_size`, default automatic; `redis_pool_timeout`, default 5 seconds). Registry writes, stats, and job pushes made from the processor's completion worker threads and the monitor thread no longer go through Sidekiq's small internal pool, which was a serialization point under load and could time out and lose completions.
+- Named processors: declare profiles with `config.processor(:llm, max_connections: 200)` and route requests with `PatientHttp::Sidekiq.execute(request, processor: :llm)`, a `processor:` option on the request itself, or `with_sidekiq_options("processor" => "llm")`. Each profile runs as an independent processor with its own capacity, timeouts, and threads, so one workload class cannot starve another. The processor name is serialized into the job arguments, so retries and crash recovery keep their routing. Jobs from older gem versions run on the `:default` processor.
+- Local stats aggregation (`stats_flush_interval`, default 5 seconds; 0 restores synchronous writes). Request metrics accumulate in memory and flush to Redis in one pipelined write per interval, removing the per-request write to the shared totals hash key. `get_totals` reports per-processor requests, duration, errors, and capacity rejections when more than one profile is configured.
+- A result that can never be delivered no longer keeps its crash-recovery record. Delivery is retried and then recovered as before, but a failure that means the result cannot be serialized (`JSON::GeneratorError` and the `Encoding` errors, in the failure or in its cause chain) is permanent: the request is counted as an `undeliverable_result` error and its job is moved to the Sidekiq dead set. Such a request previously stayed in the registry, counted as in flight, and was re-enqueued and failed again after every process restart.
+- The Web UI dashboard reports the high-water mark of requests in flight for each processor. The count only rises when a processor accepts a request, so the mark is recorded exactly rather than sampled, and it costs one comparison in memory. It is the most requests one process held at once, so it compares with `max_connections`, which is also per process. Like the other statistics it covers everything since they were last cleared.
+- The Web UI dashboard lists the requests that have been in flight the longest, with the URL, HTTP method, processor, and age of each. The details are written next to the crash-recovery record, so a request left behind by a process that died stays listed until the orphan collector re-enqueues it. The URL is sanitized first: the user name, password, query string, and fragment are removed. Use `config.inflight_url_sanitizer` to redact more, or `config.inflight_details = false` to record nothing.
+- The Web UI dashboard breaks the statistics down by processor when more than one profile is configured, reporting each processor's inflight requests, capacity, utilization, requests, errors, capacity rejections, and average duration. Each process publishes the capacity of its processors with its heartbeat, which the monitor thread now sends on every pass so the inflight counts stay current.
+- Capacity fast path: requests are rejected with a cheap in-memory capacity check before any Redis registration. A rejection previously cost three Redis round trips (register, unregister, stat); it now costs none.
+- The `completion_failed` processor event is handled by keeping the crash-recovery registry entry, so a request whose result could not be delivered is re-enqueued by the orphan collector instead of being silently lost, unless the failure is permanent (see above). Requires patient_http 1.5.0.
+- A warning is logged at startup when the hiredis Redis driver is detected, because its blocking I/O can stall the reactor thread if application code calls Redis from processor callbacks.
+
+### Changed
+
+- The orphan collector removes orphans in batches of 100 with a single Lua call per batch (invoked by `EVALSHA`), instead of one `EVAL` with the full script body per orphan. Releasing the collector's lock is now a single compare-and-delete script call instead of a WATCH/GET/MULTI sequence.
+- The shutdown re-enqueue path no longer unregisters a task twice or records a completion stat for a request that never completed.
+- The task monitor thread and stats are now shared across all processors in the process; `ProcessorObserver.new` takes `stats:` and `task_monitor:` keyword arguments.
+- The `patient_http` dependency floor is now 1.5.0.
+
+## 1.3.0
+
+### Added
+
+- Requests made in a process with a running processor now go straight to the processor instead of being enqueued through Sidekiq. The request can always be re-enqueued as a normal `RequestWorker` job, so all of the re-enqueue paths (processor shutdown, crash recovery, and the at-capacity fallback) behave the same as the enqueued path. Requests made in a `with_sidekiq_options` block always go through the Sidekiq queue, so that Sidekiq applies the options; options set with `config.sidekiq_options` do not apply to direct-executed requests because no Sidekiq job is created. The new `direct_execution` configuration option (default: `true`) turns this behavior off.
+
+### Changed
+
+- Requests are now registered in the crash-recovery registry when the processor accepts them, before the enqueue call returns, instead of when the request starts processing. A request handed to the processor is durable from that point on, and heartbeats now cover queued requests as well as in-flight ones. The entry is removed when the request completes or when a Sidekiq job owns the request again. A failure to write the registry entry rejects the request and raises to the caller, the same as a failed enqueue. This requires patient_http 1.4.0.
+
 ## 1.2.0
 
 ### Added

data/README.md

@@ -246,6 +246,52 @@ The options are applied with Sidekiq's `set` method, so any Sidekiq job option (
 
 - If the options include a `queue`, the callback job that invokes your `on_complete`/`on_error` methods is enqueued on that queue as well, so the whole request keeps one priority end to end.
 - Nested blocks merge their options, and the innermost values take precedence. The previous options are restored when the block exits, even if the block raises an error.
+- Requests made in the block always go through the Sidekiq queue, even when the processor runs in the current process, so that Sidekiq applies the options (see Direct Execution below).
+
+### Direct Execution
+
+When a request is made in a process where the processor is running (normally a Sidekiq server process), the request skips the Sidekiq queue and goes straight to the processor. This removes a round trip through Redis. The behavior is the same as the enqueued path:
+
+- The request can always be re-enqueued. It is registered in the crash-recovery registry before the call returns, so if the processor shuts down or the process crashes, the request is enqueued as a normal `RequestWorker` job. If the registry entry cannot be written (for example, Redis is unavailable), the call raises, the same as a failed enqueue.
+- If the processor is at max capacity or stops accepting requests, the request is enqueued through Sidekiq instead, and the normal Sidekiq retry behavior applies from there.
+- Requests made in a `with_sidekiq_options` block always go through the Sidekiq queue, so that Sidekiq applies the options (queue routing, scheduling, retry). Use this to route specific requests to a dedicated Sidekiq process.
+- Options set with `config.sidekiq_options` (including a `queue`) do not apply to direct-executed requests, because no Sidekiq job is created. If every request must go through the configured queue (for example, to run all requests on a dedicated Sidekiq process), set `config.direct_execution = false`.
+- Direct execution is disabled when `Sidekiq::Testing` is enabled, so tests can observe enqueued jobs as usual.
+
+You can turn this off with `config.direct_execution = false`. Do this if you route all requests to a dedicated queue with `config.sidekiq_options`, if you need Sidekiq client or server middleware to run for every request, or if you want every request to be visible as an enqueued job in Sidekiq metrics and the Web UI.
+
+### Named Processors
+
+By default all requests share one processor and one `max_connections` cap. When one process serves workload classes with very different profiles (for example, large slow API calls and small fast webhook deliveries), a burst of one class can consume all of the capacity the other class needs. Named processor profiles isolate them:
+
+```ruby
+PatientHttp::Sidekiq.configure do |config|
+  config.processor(:llm, max_connections: 200, request_timeout: 120)
+  config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+end
+```
+
+Each profile runs as an independent processor in the process, with its own capacity, timeouts, and threads. Profile options override the top-level configuration; anything not overridden (secrets, preprocessors, payload stores, encryption, logger) is shared. The `:default` processor always exists; declare `config.processor(:default, ...)` to override its options.
+
+Route a request to a processor in any of these ways:
+
+```ruby
+# Explicit option on execute
+PatientHttp::Sidekiq.execute(request, callback: MyCallback, processor: :llm)
+
+# On the request itself (survives serialization, retries, and crash recovery)
+request = PatientHttp::Request.new(:get, url, processor: :llm)
+
+# Through a request template
+template = PatientHttp::RequestTemplate.new(base_url: url, processor: :llm)
+
+# Scoped for a block
+PatientHttp::Sidekiq.with_sidekiq_options("processor" => "webhooks") do
+  PatientHttp::Sidekiq.execute(request, callback: MyCallback)
+end
+```
+
+The processor name is serialized into the job arguments, so Sidekiq retries and crash recovery keep their routing. A job that names a processor that is not configured in the executing process raises `PatientHttp::UnknownProcessorError` and goes through the normal Sidekiq retry mechanism; this makes new profile names safe to roll out gradually. Jobs enqueued by older gem versions run on the `:default` processor.
 
 ### Using Request Templates
 
@@ -464,6 +510,42 @@ PatientHttp::Sidekiq.configure do |config|
   # (use PatientHttp::Sidekiq.with_sidekiq_options to override per request)
   config.sidekiq_options = {queue: "patient_http", retry: 5}
 
+  # Whether the URL, HTTP method, and processor of each in-flight request are
+  # recorded so the Web UI can list them (default: true).
+  config.inflight_details = true
+
+  # Sanitizer applied to a URL before it is recorded. Without one, the user
+  # name, password, query string, and fragment are removed.
+  config.inflight_url_sanitizer { |url| url.sub(%r{/users/\d+}, "/users/:id") }
+
+  # Whether requests made in a process with a running processor skip the
+  # Sidekiq queue and go straight to the processor (default: true).
+  # Sidekiq options, including a queue, do not apply to direct-executed
+  # requests; set this to false to route every request through the queue.
+  config.direct_execution = true
+
+  # Size of the gem's dedicated Redis pool used by its own threads
+  # (default: nil, sized automatically from completion_threads with a
+  # floor of 10)
+  config.redis_pool_size = nil
+
+  # Checkout timeout in seconds for the dedicated Redis pool (default: 5)
+  config.redis_pool_timeout = 5
+
+  # Seconds between flushes of locally aggregated stats to Redis
+  # (default: 5; 0 writes every event synchronously)
+  config.stats_flush_interval = 5
+
+  # Number of threads that decode responses and deliver results (default: 2)
+  config.completion_threads = 2
+
+  # Maximum connections per host (default: nil, unlimited)
+  config.max_connections_per_host = 32
+
+  # Named processor profiles for workload isolation (see Named Processors)
+  config.processor(:llm, max_connections: 200, request_timeout: 120)
+  config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+
   # Handler called when a callback job exhausts all Sidekiq retries
   config.on_retries_exhausted { |error| MyAlertService.notify(error) }
 
@@ -488,6 +570,13 @@ See the [Configuration](lib/patient_http/sidekiq/configuration.rb) class for all
 - `connection_timeout`: Set this if you need to fail fast on connection establishment. Useful for detecting network issues quickly.
 - `retries`: Number of times to retry a failed request before calling the error callback.
 - `max_response_size`: Set this to limit the maximum size of HTTP responses. This helps prevent excessive memory usage from unexpectedly large responses. Responses need to be serialized to Redis as Sidekiq jobs and very large responses may cause performance issues in Redis. If a response body is text content, it will be compressed to save space in Redis. However, binary content needs to be Base64 encoded which increases size by ~33%.
+- `max_connections_per_host`: Bounds sockets per host. Verify the process file descriptor limit covers `max_connections` plus pooled idle host connections plus the application's own connections; raise the limit if needed.
+- `shutdown_timeout`: Must be below the process supervisor's termination window so the drain finishes before a hard kill. The default derives it from Sidekiq's own shutdown timeout; check any additional supervisor (container orchestrator, init system) stop timeout as well.
+- `completion_threads`: Increase when result callbacks do heavier work (serialization, encryption) and completions back up behind them.
+- `redis_pool_size`: The automatic size covers the gem's own threads. Increase it when a high request rate makes registration or completion pushes wait on checkouts.
+
+> [!WARNING]
+> Do not install `hiredis-client` in processes that run the async processor. The hiredis driver performs blocking I/O that does not yield to the fiber scheduler, so a Redis call made on the reactor thread (for example, from a custom processor observer) stalls every in-flight HTTP request. The gem logs a warning at startup when it detects the hiredis driver.
 
 > [!IMPORTANT]
 >
@@ -511,10 +600,41 @@ mount Sidekiq::Web => "/sidekiq"
 ```
 
 The Web UI shows:
-- Total requests, errors, and average duration
-- Current capacity utilization
+- Total requests, errors, average duration, and current capacity utilization
+- Per-processor capacity, utilization, requests, errors, average duration, and the high-water mark of
+  requests in flight, when more than one [named processor](#named-processors) is configured
+- The requests that have been in flight the longest, with their URL, HTTP method, processor, and age
 - Per-process inflight request counts
 
+The per-processor numbers come from the capacity each process publishes with its heartbeat, so they
+cover the processes that are currently running and can lag by a few seconds.
+
+The high-water mark is the most requests one process held on that processor at once, so compare it
+with `max_connections`, which is also per process, rather than with the capacity column, which is the
+sum across the running processes. The count only rises when a processor accepts a request, so the
+mark is exact rather than sampled. It covers everything since the statistics were last cleared.
+
+#### In-Flight Requests
+
+The URL, HTTP method, and processor of each in-flight request are recorded next to its
+crash-recovery record, and the dashboard lists the 50 oldest. A request left behind by a process
+that died stays listed until the orphan collector re-enqueues it, so this is also where you see what
+a process was working on when it stopped.
+
+The URL is sanitized before it is recorded: the user name, password, query string, and fragment are
+removed and the scheme, host, and path are kept. Paths can still carry identifiers, so you can
+redact more, or record nothing at all:
+
+```ruby
+PatientHttp::Sidekiq.configure do |config|
+  # Redact more of the URL.
+  config.inflight_url_sanitizer { |url| url.sub(%r{/users/\d+}, "/users/:id") }
+
+  # Or keep URLs out of Redis entirely.
+  config.inflight_details = false
+end
+```
+
 ### Callbacks for Custom Monitoring
 
 You can register callbacks to integrate with your monitoring system using the `after_completion` and `after_error` hooks:
@@ -588,6 +708,10 @@ The gem includes crash recovery to handle process failures:
 
 This ensures that if a Sidekiq process crashes, its in-flight requests will be retried by another process.
 
+Crash recovery gives at-least-once delivery. If a process crashes at the wrong moment (for example, between a re-enqueue and the removal of the registry entry), a request can execute more than once and its callback can fire more than once. Make your callbacks idempotent. A request is durable once the call that submits it returns; a crash during the call behaves like a failed enqueue, and the caller never received an acknowledgment.
+
+A request whose result cannot be handed to a callback job keeps its registry entry as well, so the same recovery re-enqueues it. When the failure is one that trying again cannot fix, because the result cannot be serialized, the request is not kept: it is counted as an `undeliverable_result` error and its job is moved to the Sidekiq dead set, where you can inspect it and retry it by hand.
+
 ## Testing
 
 The gem supports `Sidekiq::Testing.inline!` mode for synchronous testing. When in inline mode, async HTTP requests are executed immediately within the worker thread, blocking until completion. This allows you to write tests that verify the full request/response cycle without needing the async processor to be running.

data/VERSION

@@ -1 +1 @@
-1.2.0
+1.4.0

data/lib/patient_http/sidekiq/configuration.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "delegate"
+
 module PatientHttp
   module Sidekiq
     # Configuration for the Sidekiq integration.
@@ -24,6 +26,25 @@ module PatientHttp
       # @return [Hash, nil] Sidekiq options to apply to RequestWorker and CallbackWorker
       attr_reader :sidekiq_options
 
+      # @return [Boolean] Whether requests execute directly on a processor running in the
+      #   current process instead of being enqueued through Sidekiq
+      attr_reader :direct_execution
+
+      # @return [Boolean] Whether the URL, HTTP method, and processor of each in-flight
+      #   request are recorded so the Web UI can list them
+      attr_reader :inflight_details
+
+      # @return [Integer, nil] Size of the gem's dedicated Redis pool; nil sizes it
+      #   automatically from completion_threads
+      attr_reader :redis_pool_size
+
+      # @return [Numeric] Checkout timeout in seconds for the gem's dedicated Redis pool
+      attr_reader :redis_pool_timeout
+
+      # @return [Numeric] Seconds between flushes of locally aggregated stats to Redis
+      #   (0 flushes synchronously on every recorded event)
+      attr_reader :stats_flush_interval
+
       # @return [#call, nil] Handler invoked when a CallbackWorker job exhausts all retries.
       # @overload on_retries_exhausted
       #   Returns the current handler.
@@ -51,6 +72,8 @@ module PatientHttp
       # @param orphan_threshold [Integer] Age threshold for detecting orphaned requests in seconds
       # @param sidekiq_options [Hash, nil] Sidekiq options to apply to RequestWorker and CallbackWorker
       # @param on_retries_exhausted [#call, nil] Handler called when a CallbackWorker job exhausts retries
+      # @param direct_execution [Boolean] Whether requests execute directly on a processor
+      #   running in the current process instead of being enqueued through Sidekiq
       # @param pool_options [Hash] Options passed through to PatientHttp::Configuration.
       #   Sidekiq-aware defaults are applied for shutdown_timeout and logger
       #   if not explicitly provided.
@@ -60,6 +83,11 @@ module PatientHttp
         sidekiq_options: nil,
         payload_store_threshold: DEFAULT_PAYLOAD_STORE_THRESHOLD,
         on_retries_exhausted: nil,
+        direct_execution: true,
+        redis_pool_size: nil,
+        redis_pool_timeout: 5,
+        stats_flush_interval: 5,
+        inflight_details: true,
         **pool_options
       )
         pool_options[:shutdown_timeout] ||= (::Sidekiq.default_configuration[:timeout] || 25) - 2
@@ -68,11 +96,17 @@ module PatientHttp
         super(**pool_options)
 
         @observers = []
+        @processor_profiles = {default: {}}
         self.sidekiq_options = sidekiq_options
         self.heartbeat_interval = heartbeat_interval
         self.orphan_threshold = orphan_threshold
         self.payload_store_threshold = payload_store_threshold || DEFAULT_PAYLOAD_STORE_THRESHOLD
         self.on_retries_exhausted = on_retries_exhausted
+        self.direct_execution = direct_execution
+        self.redis_pool_size = redis_pool_size
+        self.redis_pool_timeout = redis_pool_timeout
+        self.stats_flush_interval = stats_flush_interval
+        self.inflight_details = inflight_details
       end
 
       # Set the on_retries_exhausted handler.
@@ -144,6 +178,179 @@ module PatientHttp
         apply_sidekiq_options(options)
       end
 
+      # Set whether requests execute directly on a processor running in the current
+      # process. When enabled, requests made in a process with a running processor
+      # skip the Sidekiq queue and go straight to the processor. The value is
+      # coerced to a boolean.
+      #
+      # @param value [Boolean] true to enable direct execution
+      # @return [void]
+      def direct_execution=(value)
+        @direct_execution = !!value
+      end
+
+      # Check if direct execution is enabled.
+      #
+      # @return [Boolean]
+      def direct_execution?
+        @direct_execution
+      end
+
+      # Set the size of the gem's dedicated Redis pool.
+      #
+      # @param value [Integer, nil] pool size; nil sizes the pool automatically
+      # @return [void]
+      def redis_pool_size=(value)
+        if value.nil?
+          @redis_pool_size = nil
+          return
+        end
+
+        validate_positive_integer(:redis_pool_size, value)
+        @redis_pool_size = value
+      end
+
+      # Set the checkout timeout for the gem's dedicated Redis pool.
+      #
+      # @param value [Numeric] timeout in seconds
+      # @return [void]
+      def redis_pool_timeout=(value)
+        unless value.is_a?(Numeric) && value.positive?
+          raise ArgumentError.new("redis_pool_timeout must be a positive number, got: #{value.inspect}")
+        end
+
+        @redis_pool_timeout = value
+      end
+
+      # Set the flush interval for locally aggregated stats.
+      #
+      # @param value [Numeric] seconds between flushes; 0 flushes synchronously on
+      #   every recorded event
+      # @return [void]
+      def stats_flush_interval=(value)
+        unless value.is_a?(Numeric) && value >= 0
+          raise ArgumentError.new("stats_flush_interval must be a non-negative number, got: #{value.inspect}")
+        end
+
+        @stats_flush_interval = value
+      end
+
+      # Set whether the details of each in-flight request are recorded.
+      #
+      # The URL (see +inflight_url_sanitizer+), HTTP method, and processor name are
+      # written next to the crash-recovery record so the Web UI can list the requests
+      # that are in flight. Turn this off to keep URLs out of Redis entirely.
+      #
+      # @param value [Boolean] true to record the details
+      # @return [void]
+      def inflight_details=(value)
+        @inflight_details = !!value
+      end
+
+      # Check if in-flight request details are recorded.
+      #
+      # @return [Boolean]
+      def inflight_details?
+        @inflight_details
+      end
+
+      # Set the sanitizer applied to a request URL before it is recorded for the
+      # Web UI, or read the current one.
+      #
+      # The sanitizer receives the full URL and returns what to display. Without
+      # one, the user name, password, query string, and fragment are removed and
+      # the scheme, host, and path are kept. Use it to redact more, such as an
+      # identifier in the path.
+      #
+      # @example
+      #   config.inflight_url_sanitizer { |url| url.sub(%r{/users/\d+}, "/users/:id") }
+      #
+      # @overload inflight_url_sanitizer
+      #   @return [#call, nil] the current sanitizer
+      # @overload inflight_url_sanitizer(&block)
+      #   @yield [url] block that returns the URL to display
+      #   @yieldparam url [String] the full request URL
+      def inflight_url_sanitizer(&block)
+        if block
+          @inflight_url_sanitizer = block
+        else
+          @inflight_url_sanitizer
+        end
+      end
+
+      # Set the sanitizer applied to a request URL before it is recorded.
+      #
+      # @param value [#call, nil] a callable object, or nil to use the default
+      # @raise [ArgumentError] If value is not callable and not nil
+      def inflight_url_sanitizer=(value)
+        if value && !value.respond_to?(:call)
+          raise ArgumentError.new("inflight_url_sanitizer must respond to #call, got: #{value.class}")
+        end
+
+        @inflight_url_sanitizer = value
+      end
+
+      # Declare a named processor profile, or read one back.
+      #
+      # Each profile becomes an independent processor with its own capacity,
+      # timeouts, and threads. Options are overrides applied on top of this
+      # configuration's HTTP pool options. Requests select a processor with the
+      # +processor:+ option on +PatientHttp::Sidekiq.execute+ (or on the
+      # request itself). The +:default+ profile always exists; declaring it
+      # overrides options for the default processor.
+      #
+      # @example
+      #   PatientHttp::Sidekiq.configure do |config|
+      #     config.processor(:llm, max_connections: 200, request_timeout: 120)
+      #     config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+      #   end
+      #
+      # @param name [Symbol, String] the processor name
+      # @param options [Hash] overrides for PatientHttp::Configuration options
+      # @return [Hash] the stored options for the profile
+      def processor(name, **options)
+        key = normalize_processor_name(name)
+
+        if options.any?
+          validate_profile_options!(options)
+          @processor_profiles[key] = options
+        end
+
+        @processor_profiles[key]
+      end
+
+      # All declared processor profiles. Always includes :default.
+      #
+      # @return [Hash{Symbol => Hash}] profile options by processor name
+      def processor_profiles
+        @processor_profiles.dup
+      end
+
+      # Whether more than one processor profile is declared.
+      #
+      # @return [Boolean]
+      def multiple_processors?
+        @processor_profiles.size > 1
+      end
+
+      # Build the effective configuration for a named processor. The default
+      # profile with no overrides is this configuration itself; other profiles
+      # get a view of this configuration with their overrides applied, so they
+      # share secrets, preprocessors, payload stores, and encryption.
+      #
+      # @param name [Symbol, String] the processor name
+      # @return [PatientHttp::Configuration] the configuration for the processor
+      # @raise [ArgumentError] if the profile is not declared
+      def processor_config(name)
+        key = normalize_processor_name(name)
+        profile = @processor_profiles[key]
+        raise ArgumentError.new("Unknown processor profile: #{name.inspect}") unless profile
+
+        return self if profile.empty?
+
+        ProfileConfiguration.new(self, profile)
+      end
+
       # Convert to hash for inspection
       # @return [Hash] hash representation with string keys
       def to_h
@@ -152,12 +359,62 @@ module PatientHttp
           "heartbeat_interval" => heartbeat_interval,
           "orphan_threshold" => orphan_threshold,
           "sidekiq_options" => sidekiq_options,
-          "on_retries_exhausted" => on_retries_exhausted ? "defined" : nil
+          "direct_execution" => direct_execution,
+          "on_retries_exhausted" => on_retries_exhausted ? "defined" : nil,
+          "redis_pool_size" => redis_pool_size,
+          "redis_pool_timeout" => redis_pool_timeout,
+          "stats_flush_interval" => stats_flush_interval,
+          "inflight_details" => inflight_details,
+          "processor_profiles" => processor_profiles.keys.map(&:to_s)
         )
       end
 
+      # View of a base configuration with a profile's option overrides applied.
+      # Everything not overridden (secrets, preprocessors, payload stores,
+      # logging) delegates to the base configuration, so all processors share
+      # those registries.
+      #
+      # Overrides are applied through a real PatientHttp::Configuration so that
+      # each option's writer performs its own normalization. Readers whose
+      # value a writer derives from an overridden option are taken from that
+      # configuration as well, so an override cannot be half applied.
+      class ProfileConfiguration < SimpleDelegator
+        # Readers populated as a side effect of another option's writer.
+        DERIVED_READERS = {
+          encryption_key: [:encryption, :decryption, :encryptor]
+        }.freeze
+
+        def initialize(base_configuration, overrides)
+          super(base_configuration)
+          normalized = PatientHttp::Configuration.new(**overrides)
+          readers = overrides.keys.flat_map { |key| [key.to_sym, *DERIVED_READERS[key.to_sym]] }.uniq
+          readers.each do |reader|
+            next unless normalized.respond_to?(reader)
+
+            define_singleton_method(reader) do |*args, &block|
+              normalized.public_send(reader, *args, &block)
+            end
+          end
+        end
+      end
+
       private
 
+      # Profile options must be valid PatientHttp::Configuration options. A
+      # throwaway configuration exercises each option's own validation.
+      def validate_profile_options!(options)
+        PatientHttp::Configuration.new(**options)
+      rescue ArgumentError => e
+        raise ArgumentError.new("Invalid processor profile options: #{e.message}")
+      end
+
+      def normalize_processor_name(name)
+        key = name.to_s
+        raise ArgumentError.new("processor name cannot be empty") if key.empty?
+
+        key.to_sym
+      end
+
       def apply_sidekiq_options(options)
         PatientHttp::Sidekiq::RequestWorker.sidekiq_options(options)
         PatientHttp::Sidekiq::CallbackWorker.sidekiq_options(options)

data/lib/patient_http/sidekiq/direct_task_handler.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module PatientHttp
+  module Sidekiq
+    # TaskHandler for requests executed directly on the local processor
+    # without an enqueued Sidekiq job. Requests with scoped Sidekiq options
+    # always go through the queue, so the handler only deals with the
+    # default RequestWorker options.
+    #
+    # Retry enqueues a normal RequestWorker job with the original arguments,
+    # so the fail-back behavior matches the enqueued path. The sidekiq_job
+    # hash is a minimal job record kept for the crash-recovery registry; it
+    # has no jid because no Sidekiq job exists until the request is
+    # re-enqueued, so job_id returns nil.
+    class DirectTaskHandler < TaskHandler
+      # @param args [Array] the RequestWorker job arguments
+      def initialize(args)
+        @args = args
+        super(minimal_job_record)
+      end
+
+      # Re-enqueue the request as a normal RequestWorker job.
+      #
+      # @return [String] the job ID
+      def retry
+        PatientHttp::Sidekiq.with_redis_pool do
+          RequestWorker.perform_async(*@args)
+        end
+      end
+
+      private
+
+      # Minimal pushable job record for the crash-recovery registry.
+      # TaskMonitor serializes it to Redis and the orphan GC pushes it
+      # verbatim, possibly from another process, so it cannot enqueue
+      # through this handler. The worker options are included because
+      # Sidekiq::Client.push does not apply them when "class" is a String.
+      #
+      # @return [Hash]
+      def minimal_job_record
+        RequestWorker.get_sidekiq_options
+          .merge("class" => RequestWorker.name, "args" => @args)
+      end
+    end
+  end
+end