patient_http-sidekiq 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5d2f470219dae6e479974adecec97231a42918686c415b3d601c1803274cdb06
+  data.tar.gz: b8af3ece14b9c58032153ee2870c7122ae30b460894aad31bbd018f879e79da3
+SHA512:
+  metadata.gz: 9f78b1ade928fc188d3e48247fe6a6a2a3891470b820b24e912737cdea8ad8ce1bbb3b0324e3dc0ab7417cf0c5142db9045b3ee24064e7d4d54de45ea191b126
+  data.tar.gz: 8050fbbbaa96eb45217ef7df4c1fb554f3b79fe8440156ed09e31e42d290fdc5cf849ebf20be2eb03dcf4b35a72b827d740e41606412cd1a987ea60ce1333b87

data/ARCHITECTURE.md

@@ -0,0 +1,496 @@
+# Architecture
+
+## Overview
+
+PatientHttp::Sidekiq provides a Sidekiq integration layer for the [patient_http](https://github.com/bdurand/patient_http) gem, enabling long-running HTTP requests to be offloaded from Sidekiq worker threads to a dedicated async I/O processor. This integration uses Sidekiq's job system for request enqueueing and callback invocation, while leveraging patient_http's Fiber-based concurrency to handle hundreds of concurrent HTTP requests without blocking worker threads.
+
+## Key Design Principles
+
+1. **Non-blocking Workers**: Worker threads enqueue HTTP requests via Sidekiq jobs and immediately return, freeing them to process other jobs
+2. **Singleton Processor**: One async I/O processor (from patient_http) per Sidekiq process handles all HTTP requests using Fiber-based concurrency
+3. **Callback Service Pattern**: HTTP responses are processed by callback service classes with `on_complete` and `on_error` methods, invoked via Sidekiq jobs
+4. **Lifecycle Integration**: Processor lifecycle is tightly coupled with Sidekiq's startup, quiet, and shutdown events
+5. **Sidekiq-Native Task Handling**: Request lifecycle operations (enqueueing, callbacks, retries) use Sidekiq's job system
+
+## Core Components
+
+### PatientHttp::Processor (from patient_http gem)
+The heart of the system - a singleton that runs in a dedicated thread with its own Fiber reactor. Manages the async HTTP request queue and handles concurrent request execution using Ruby's `async` gem with HTTP/2 connection pooling.
+
+### TaskHandler
+Implements the patient_http's `TaskHandler` interface to integrate with Sidekiq's job system:
+- Enqueues `CallbackWorker` jobs when requests complete or error
+- Handles request retries via `Sidekiq::Client.push`
+- Manages large payloads via `ExternalStorage` before enqueueing
+
+### RequestWorker
+A Sidekiq worker that receives HTTP request specifications and submits them to the async processor. This allows HTTP requests to be made from anywhere in your code (Rails controllers, background jobs, rake tasks, etc.) by enqueueing a job.
+
+### CallbackWorker
+A Sidekiq worker that invokes callback service methods (`on_complete` or `on_error`) when HTTP requests complete. Handles payload decryption and external storage retrieval.
+
+### LifecycleHooks
+Registers Sidekiq server lifecycle hooks to automatically:
+- Start the processor when Sidekiq starts (`:startup` event)
+- Drain the processor when Sidekiq receives TSTP signal (`:quiet` event)
+- Stop the processor gracefully when Sidekiq shuts down (`:shutdown` event)
+
+### RequestHelper Handler Registration
+On startup, the integration automatically registers a handler via `PatientHttp.register_handler` so that classes including the `RequestHelper` module can use `async_get`, `async_post`, etc. The handler translates those calls into `PatientHttp::Sidekiq.execute` invocations. The handler is unregistered on shutdown.
+
+### ProcessorObserver
+Observes processor state changes and updates the TaskMonitor's Redis heartbeats, enabling distributed crash recovery.
+
+### TaskMonitor
+Manages crash recovery by tracking in-flight requests in Redis:
+- Maintains a sorted set of request IDs indexed by timestamp
+- Stores request payloads with metadata for recovery
+- Detects orphaned requests when processes crash
+- Re-enqueues orphaned requests via Sidekiq
+
+### TaskMonitorThread
+Background thread that periodically:
+- Updates heartbeat timestamps for in-flight requests
+- Scans for orphaned requests from crashed processes
+- Performs garbage collection on stale Redis data
+
+### Request/Response/Error (from patient_http gem)
+Immutable value objects representing HTTP requests and their results. All are JSON-serializable for passing through Sidekiq jobs.
+
+### ExternalStorage (from patient_http gem)
+Handles storage and retrieval of large payloads (requests, responses, errors) to Redis or disk when they exceed the payload size threshold, preventing Sidekiq job serialization issues.
+
+### Configuration
+Sidekiq-specific configuration including:
+- Callback queue names
+- Encryption for sensitive data
+- External storage settings
+- Integration with patient_http's configuration
+
+## Callback Service Pattern
+
+When HTTP requests complete, the processor enqueues CallbackWorker jobs to invoke the appropriate callback service method:
+
+- **Success callbacks**: The `on_complete` method receives a `Response` object with status, headers, body, and callback arguments
+- **Error callbacks**: The `on_error` method receives an `Error` object with error details and callback arguments
+- **Callback arguments** are passed via the `callback_args:` option and accessed via `response.callback_args[:key]` or `error.callback_args[:key]`
+
+Example:
+```ruby
+# Define a callback service class
+class FetchDataCallback
+  def on_complete(response)
+    user_id = response.callback_args[:user_id]
+    User.find(user_id).update!(data: response.json)
+  end
+
+  def on_error(error)
+    user_id = error.callback_args[:user_id]
+    Rails.logger.error("Failed to fetch data for user #{user_id}: #{error.message}")
+  end
+end
+
+# Make a request from anywhere in your code
+PatientHttp::Sidekiq.get(
+  "https://api.example.com/users/123",
+  callback: FetchDataCallback,
+  callback_args: {user_id: 123}
+)
+```
+
+## Request Lifecycle
+
+```mermaid
+sequenceDiagram
+    participant App as Application Code
+    participant Module as PatientHttp::Sidekiq
+    participant ReqWorker as RequestWorker
+    participant Processor as PatientHttp::Processor
+    participant Sidekiq as Sidekiq Queue
+    participant Handler as TaskHandler
+    participant CbWorker as CallbackWorker
+    participant Callback as Callback Service
+
+    App->>Module: get(url, callback: MyCallback)
+    Module->>Sidekiq: Enqueue RequestWorker
+    Sidekiq->>ReqWorker: Execute job
+    ReqWorker->>Processor: submit(request, handler)
+    activate Processor
+    Note over Processor: Request queued<br/>in memory
+    Processor-->>ReqWorker: Returns immediately
+    ReqWorker-->>Sidekiq: Job completes
+    deactivate Processor
+
+    Note over Sidekiq: Worker thread free<br/>to process other jobs
+
+    activate Processor
+    Processor->>Processor: Fiber reactor<br/>dequeues request
+    Processor->>Processor: Execute HTTP request<br/>(non-blocking with async)
+
+    alt HTTP Request Completes
+        Processor->>Handler: on_complete(response, callback)
+        Handler->>Sidekiq: Enqueue CallbackWorker
+        Sidekiq->>CbWorker: Execute job
+        CbWorker->>Callback: on_complete(response)
+        Callback->>Callback: Process response
+    else Error Raised
+        Processor->>Handler: on_error(error, callback)
+        Handler->>Sidekiq: Enqueue CallbackWorker
+        Sidekiq->>CbWorker: Execute job
+        CbWorker->>Callback: on_error(error)
+        Callback->>Callback: Handle error
+    end
+    deactivate Processor
+```
+
+Key integration points:
+1. **RequestWorker** converts Sidekiq job args into patient_http Request objects
+2. **TaskHandler** converts processor callbacks into Sidekiq jobs
+3. **CallbackWorker** invokes the user's callback service methods
+4. **ExternalStorage** handles large payloads transparently at each step
+
+## Component Relationships
+
+```mermaid
+erDiagram
+    PATIENT-HTTP-SIDEKIQ ||--|| PATIENT-HTTP : "integrates with"
+    PATIENT-HTTP-SIDEKIQ ||--|| SIDEKIQ-TASK-HANDLER : "provides"
+    PATIENT-HTTP-SIDEKIQ ||--|| REQUEST-WORKER : "defines"
+    PATIENT-HTTP-SIDEKIQ ||--|| CALLBACK-WORKER : "defines"
+    PATIENT-HTTP-SIDEKIQ ||--|| TASK-MONITOR : "manages"
+    PATIENT-HTTP-SIDEKIQ ||--|| LIFECYCLE-HOOKS : "registers"
+
+    PATIENT-HTTP ||--|| PROCESSOR : "provides"
+    PATIENT-HTTP ||--|| REQUEST : "defines"
+    PATIENT-HTTP ||--|| RESPONSE : "defines"
+    PATIENT-HTTP ||--|| ERROR : "defines"
+    PATIENT-HTTP ||--|| EXTERNAL-STORAGE : "provides"
+
+    PROCESSOR ||--|| TASK-HANDLER : "uses"
+    PROCESSOR ||--o{ REQUEST : "processes"
+
+    SIDEKIQ-TASK-HANDLER ||--|| CALLBACK-WORKER : "enqueues"
+    SIDEKIQ-TASK-HANDLER ||--|| EXTERNAL-STORAGE : "uses"
+
+    REQUEST-WORKER ||--|| PROCESSOR : "submits to"
+    REQUEST-WORKER ||--|| SIDEKIQ-TASK-HANDLER : "creates"
+
+    CALLBACK-WORKER ||--|| CALLBACK-SERVICE : "invokes"
+    CALLBACK-WORKER ||--o| RESPONSE : "receives"
+    CALLBACK-WORKER ||--o| ERROR : "receives"
+
+    TASK-MONITOR ||--|| TASK-MONITOR-THREAD : "runs"
+    TASK-MONITOR ||--|| REDIS : "tracks in"
+
+    LIFECYCLE-HOOKS ||--|| PROCESSOR : "controls"
+
+    PROCESSOR {
+        string state
+        int queue_size
+        thread reactor_thread
+    }
+
+    REQUEST {
+        string http_method
+        string url
+        hash headers
+        string body
+        float timeout
+    }
+
+    RESPONSE {
+        int status
+        hash headers
+        string body
+        hash callback_args
+    }
+
+    ERROR {
+        string message
+        string error_class
+        hash callback_args
+    }
+
+    CALLBACK-SERVICE {
+        method on_complete
+        method on_error
+    }
+
+    TASK-MONITOR {
+        hash inflight_tasks
+        string redis_key
+    }
+```
+
+### Integration Points
+
+**Sidekiq → patient_http:**
+- `RequestWorker` converts Sidekiq job args to `PatientHttp::Request` objects
+- `TaskHandler` implements `PatientHttp::TaskHandler` interface
+- `Processor` is created and managed by the Sidekiq integration layer
+
+**patient_http → Sidekiq:**
+- Processor calls `TaskHandler#on_complete` and `TaskHandler#on_error` callbacks
+- `TaskHandler` enqueues `CallbackWorker` jobs via Sidekiq
+- Large payloads are stored via `ExternalStorage` before enqueueing
+
+## Process Model
+
+Each Sidekiq process runs:
+- Multiple worker threads (configured via Sidekiq concurrency)
+- **One** async HTTP processor thread (from patient_http)
+- **One** fiber reactor within the processor thread
+- **One** task monitor thread for crash recovery
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Sidekiq Process                          │
+│                                                             │
+│  ┌──────────────┐   ┌──────────────┐  ┌──────────────┐      │
+│  │ Worker       │   │ Worker       │  │ Worker       │      │
+│  │ Thread 1     │   │ Thread 2     │  │ Thread N     │      │
+│  │              │   │              │  │              │      │
+│  │ Executes:    │   │ Executes:    │  │ Executes:    │      │
+│  │ - RequestW.  │   │ - CallbackW. │  │ - Other Jobs │      │
+│  └──────┬───────┘   └──────┬───────┘  └──────┬───────┘      │
+│         │                  │                 │              │
+│         └──────────────────┼─────────────────┘              │
+│                            │                                │
+│                            ▼                                │
+│               ┌─────────────────────────┐                   │
+│               │  PatientHttp          │                   │
+│               │  Processor              │                   │
+│               │  (Dedicated Thread)     │                   │
+│               │                         │                   │
+│               │  ┌───────────────────┐  │                   │
+│               │  │  Async Fiber      │  │                   │
+│               │  │  Reactor          │  │                   │
+│               │  │  ═════════════    │  │                   │
+│               │  │  - HTTP/2 pools   │  │                   │
+│               │  │  - 100+ concurrent│  │                   │
+│               │  │    requests       │  │                   │
+│               │  │  - Non-blocking   │  │                   │
+│               │  │    I/O            │  │                   │
+│               │  └───────────────────┘  │                   │
+│               └─────────────────────────┘                   │
+│                            │                                │
+│               ┌────────────┴─────────────┐                  │
+│               │  TaskMonitorThread       │                  │
+│               │  (Crash Recovery)        │                  │
+│               │                          │                  │
+│               │  - Heartbeat updates     │                  │
+│               │  - Orphan detection      │                  │
+│               │  - Redis GC              │                  │
+│               └──────────────────────────┘                  │
+└─────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+                    ┌───────────────┐
+                    │     Redis     │
+                    │               │
+                    │  - Job queues │
+                    │  - Inflight   │
+                    │    tracking   │
+                    │  - Payloads   │
+                    └───────────────┘
+```
+
+### Architectural Layers
+
+**Application Layer:**
+- User code calls `PatientHttp::Sidekiq.get/post/etc`
+- Or includes `PatientHttp::RequestHelper` for `async_get/async_post/etc` instance methods
+- Callback services implement `on_complete` and `on_error`
+
+**Sidekiq Integration Layer (this gem):**
+- `RequestWorker` - Sidekiq job to submit requests
+- `CallbackWorker` - Sidekiq job to invoke callbacks
+- `TaskHandler` - Bridges processor callbacks to Sidekiq jobs
+- `LifecycleHooks` - Manages processor lifecycle
+- `TaskMonitor` - Crash recovery and inflight tracking
+
+**HTTP Processing Layer (patient_http):**
+- `Processor` - Main async I/O processor
+- `Request/Response/Error` - Value objects
+- `ExternalStorage` - Large payload handling
+- Async fiber scheduler and HTTP/2 connection pools
+
+## Concurrency Model
+
+The system uses multiple levels of concurrency:
+
+### Sidekiq Worker Threads
+- Process Sidekiq jobs from queues
+- Execute `RequestWorker` and `CallbackWorker` jobs
+- Block only briefly while submitting requests to the processor
+
+### Async HTTP Processor Thread (from patient_http)
+- Runs Ruby's Fiber scheduler (`async` gem) for non-blocking I/O
+- Maintains HTTP/2 connection pools for efficient connection reuse
+- Multiplexes hundreds of concurrent HTTP requests via fibers
+- Each HTTP request runs in its own fiber (lightweight concurrency)
+
+### Task Monitor Thread
+- Periodically updates Redis heartbeats for in-flight requests
+- Scans for orphaned requests from crashed processes
+- Performs garbage collection on stale data
+
+**Benefits:**
+1. **Worker threads remain free** - submitting a request to the processor takes ~1ms
+2. **Fiber-based multiplexing** - handle hundreds of concurrent requests in a single thread
+3. **HTTP/2 connection reuse** - multiple requests share persistent connections
+4. **Non-blocking I/O** - fibers yield during network I/O, allowing other requests to progress
+
+## State Management
+
+The processor (from patient_http) maintains state through its lifecycle, managed by Sidekiq lifecycle hooks:
+
+- **stopped**: Initial state, not processing requests
+- **starting**: Processor is initializing, reactor thread launching
+- **running**: Actively processing requests
+- **draining**: Not accepting new requests (triggered by Sidekiq's `:quiet` event), completing in-flight
+- **stopping**: Shutting down (triggered by Sidekiq's `:shutdown` event), waiting for requests to finish
+
+**Lifecycle Integration:**
+
+```ruby
+# Registered automatically via LifecycleHooks
+Sidekiq.configure_server do |config|
+  config.on(:startup) { PatientHttp::Sidekiq.start }    # → processor state: running
+  config.on(:quiet)   { PatientHttp::Sidekiq.quiet }    # → processor state: draining
+  config.on(:shutdown) { PatientHttp::Sidekiq.stop }    # → processor state: stopping
+end
+```
+
+## Crash Recovery
+
+In-flight requests are tracked in Redis to enable recovery when Sidekiq processes crash:
+
+### TaskMonitor
+- Maintains a Redis sorted set of in-flight request IDs indexed by timestamp
+- Stores request payloads with metadata (Sidekiq job, callback info)
+- Each process has a unique process ID: `hostname:pid:hex`
+
+### TaskMonitorThread
+- Runs in background, periodically updating heartbeat timestamps in Redis
+- Scans for orphaned requests (no heartbeat update within threshold)
+- Re-enqueues orphaned requests via `Sidekiq::Client.push`
+
+### Recovery Process
+1. `ProcessorObserver` notifies `TaskMonitor` when requests start/complete
+2. `TaskMonitorThread` updates heartbeat timestamps in Redis
+3. If a process crashes, heartbeat updates stop
+4. Other processes' monitor threads detect stale timestamps
+5. Orphaned requests are atomically removed and re-enqueued
+6. Prevents lost work during deployments or crashes
+
+**Redis Keys:**
+- `sidekiq:patient_http:inflight_index` - Sorted set of request IDs by timestamp
+- `sidekiq:patient_http:inflight_jobs` - Hash of request payloads
+- `sidekiq:patient_http:processes` - Set of active process IDs
+- `sidekiq:patient_http:gc_lock` - Distributed lock for garbage collection
+- `sidekiq:patient_http:gc_last_run` - Timestamp of last garbage collection run
+
+## Configuration
+
+Configuration is split between Sidekiq-specific concerns and patient_http settings:
+
+### Sidekiq Integration Settings
+```ruby
+PatientHttp::Sidekiq.configure do |config|
+  # Sidekiq worker options (applied to both RequestWorker and CallbackWorker)
+  config.sidekiq_options = {queue: "patient_http", retry: 5}
+
+  # Encryption (for sensitive data in Sidekiq jobs; inherited from PatientHttp::Configuration)
+  config.encryption_key = ENV["PATIENT_HTTP_ENCRYPTION_KEY"]
+
+  # External storage threshold (for large payloads)
+  config.payload_store_threshold = 100_000   # bytes
+
+  # Shutdown timeout
+  config.shutdown_timeout = 30               # seconds
+end
+```
+
+### Async HTTP Pool Settings (delegated)
+All patient_http configuration is accessible:
+```ruby
+PatientHttp::Sidekiq.configure do |config|
+  # HTTP settings
+  config.request_timeout = 30
+  config.max_connections = 100
+
+  # Retry behavior
+  config.retries = 3
+
+  # Proxy settings
+  config.proxy_url = ENV["HTTP_PROXY"]
+end
+```
+
+The configuration object is passed to the `PatientHttp::Processor` on startup.
+
+## Web UI
+
+Optional Sidekiq Web integration (via `WebUI` module) provides:
+
+- Real-time processor state (running/draining/stopped)
+- In-flight request counts by process
+- Historical statistics from patient_http metrics
+- Health indicators
+- Max connection capacity per process
+
+The Web UI reads from:
+- `TaskMonitor` Redis keys for inflight counts
+- `PatientHttp::Processor` stats for metrics
+- Process registry for distributed monitoring
+
+## Data Flow
+
+### Making a Request
+
+```
+Application Code
+  ↓ PatientHttp::Sidekiq.get(url, callback: MyCallback)
+RequestWorker job enqueued
+  ↓ Sidekiq processes job
+RequestWorker#perform
+  ↓ Creates Request, TaskHandler
+PatientHttp::Processor.submit(request, handler)
+  ↓ Queued in memory
+TaskMonitor.track(request_id, handler)
+  ↓ Stored in Redis
+Fiber reactor processes request
+  ↓ Non-blocking HTTP I/O
+Response/Error received
+```
+
+### Processing a Response
+
+```
+Fiber completes with Response
+  ↓
+TaskHandler#on_complete(response, callback)
+  ↓ Stores via ExternalStorage if large
+CallbackWorker job enqueued
+  ↓ Sidekiq processes job
+CallbackWorker#perform
+  ↓ Fetches from ExternalStorage if needed
+  ↓ Decrypts payload
+MyCallback.new.on_complete(response)
+  ↓ User code executes
+TaskMonitor.untrack(request_id)
+  ↓ Removed from Redis
+```
+
+## Thread Safety
+
+- **Thread-safe submission**: `Processor` uses thread-safe queues for request submission
+- **Atomic state changes**: Processor state managed with atomic operations
+- **Redis-based coordination**: TaskMonitor uses Redis for distributed coordination
+- **Immutable values**: Request/Response/Error objects are immutable once created
+- **Sidekiq job isolation**: Each CallbackWorker runs in its own thread
+
+## Further Reading
+
+- [README](README.md)

data/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## 1.0.0
+
+### Added
+
+- Dedicated async HTTP processor thread for Sidekiq to avoid blocking worker threads during in-flight requests.
+- `PatientHttp::Sidekiq` API with convenience methods for common HTTP verbs (`get`, `post`, `put`, `patch`, and `delete`).
+- Callback-based completion and error handling via `on_complete` and `on_error`, executed as Sidekiq jobs.
+- Support for callback context via `callback_args`, available from response and error objects.
+- Built-in runtime visibility with task monitoring and a Sidekiq Web UI page for async HTTP activity.
+- Crash/failure recovery with heartbeat-based orphan detection and automatic re-enqueue of interrupted requests.

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2026 Brian Durand
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.