patient_http-solid_queue 1.2.1 → 1.3.0

data/README.md

@@ -6,13 +6,13 @@
 
 *Built for APIs that like to think.*
 
-This gem provides a mechanism to offload HTTP requests to a dedicated async I/O processor running in your Solid Queue worker process using the [patient_http gem](https://github.com/bdurand/patient_http). Worker threads are freed immediately while HTTP requests are in flight so that they can do other work instead of waiting for HTTP responses.
+This gem runs HTTP requests from Solid Queue on a dedicated async I/O processor in your Solid Queue worker process, using the [patient_http gem](https://github.com/bdurand/patient_http). Worker threads don't wait for HTTP responses, so they're free to run other jobs while requests are in flight.
 
 ## Motivation
 
-Solid Queue is designed with the assumption that jobs are short-lived and complete quickly. Long-running HTTP requests block worker threads from processing other jobs, leading to increased latency and reduced throughput. This is particularly problematic when calling LLM or AI APIs, where requests can take many seconds to complete.
+Solid Queue works best when jobs finish quickly. A long HTTP request blocks a worker thread, so other jobs wait, latency rises, and throughput drops. LLM and other AI APIs make this worse, because a request can take many seconds to finish.
 
-**The Problem:**
+Without this gem, each slow request holds a worker thread for its full duration:
 
 ```
 ┌────────────────────────────────────────────────────────────────────────┐
@@ -26,7 +26,7 @@ Solid Queue is designed with the assumption that jobs are short-lived and comple
 └────────────────────────────────────────────────────────────────────────┘
 ```
 
-**The Solution:**
+With this gem, a worker thread only hands off the request, and the async processor waits for the response:
 
 ```
 ┌────────────────────────────────────────────────────────────────────────┐
@@ -42,11 +42,31 @@ Solid Queue is designed with the assumption that jobs are short-lived and comple
 └────────────────────────────────────────────────────────────────────────┘
 ```
 
-The async processor runs in a dedicated thread within your Solid Queue worker process, using Ruby's Fiber-based concurrency to handle hundreds of concurrent HTTP requests without blocking. When an HTTP request completes, a callback service is invoked for processing.
+The async processor runs in a dedicated thread in your Solid Queue worker process. It uses Ruby's fiber-based concurrency to run hundreds of HTTP requests at the same time without blocking. When a request finishes, the gem calls your callback service.
 
-## Quick Start
+## Quick start
 
-### 1. Create a Callback Service
+### 1. Install the gem
+
+Add the gem to your Gemfile:
+
+```ruby
+gem "patient_http-solid_queue"
+```
+
+Then install it, run the install generator, and run the migration:
+
+```bash
+bundle install
+bin/rails generate patient_http:solid_queue:install
+bin/rails db:migrate
+```
+
+The generator creates the migration for the crash-recovery tables and a commented initializer to start from. For details, including multi-database setups, see [Installation](#installation).
+
+No other setup is required. When the gem loads, it registers the request handler and connects the processor to Solid Queue's startup and shutdown. You don't need to call a setup method. Every option has a working default. To change the defaults, see [Configuration](#configuration).
+
+### 2. Create a callback service
 
 Define a callback service class with `on_complete` and `on_error` methods:
 
@@ -69,9 +89,9 @@ class FetchDataCallback
 end
 ```
 
-### 2. Make HTTP Requests
+### 3. Make HTTP requests
 
-Make HTTP requests from anywhere in your code using `PatientHttp`:
+Make HTTP requests from anywhere in your code with the `PatientHttp` module:
 
 ```ruby
 PatientHttp.get(
@@ -82,32 +102,80 @@ PatientHttp.get(
 )
 ```
 
-### 3. That's It!
+The gem enqueues the request as an Active Job, which runs the request on a [PatientHttp](https://github.com/bdurand/patient_http) processor. When the request finishes, another Active Job calls your callback's `on_complete` method. If the request fails, the job calls `on_error` instead.
 
-The request will be enqueued as an Active Job and passed to a [PatientHttp](https://github.com/bdurand/patient_http) processor to execute asynchronously. When the HTTP request completes, your callback's `on_complete` method is executed in another Active Job.
+The `response.callback_args` and `error.callback_args` methods return the arguments that you passed with the `callback_args` option.
 
-If an error occurs during the request, the `on_error` method is called instead.
+For other HTTP methods, use `PatientHttp.post`, `PatientHttp.put`, `PatientHttp.patch`, `PatientHttp.delete`, `PatientHttp.head`, and `PatientHttp.query`. For the full API reference, see the [patient_http documentation](https://github.com/bdurand/patient_http).
 
-You can also call `PatientHttp.post`, `PatientHttp.put`, `PatientHttp.patch`, and `PatientHttp.delete` for other HTTP methods. See the [patient_http docs](https://github.com/bdurand/patient_http) for the full API reference.
+> [!IMPORTANT]
+> Don't raise an error in `on_error` to retry the request. Active Job retries the callback job, not the request. To retry the request, make a new request from `on_error`. Make sure that the retries stop if the error persists, or they can loop forever.
+>
+> The `on_error` callback runs only when the request raises an exception, such as a timeout or a connection failure. By default, HTTP error status codes (4xx and 5xx) don't call `on_error`. The gem treats these responses as completed requests and passes them to `on_complete`. To treat HTTP errors as exceptions, see [Handle HTTP error responses](#handle-http-error-responses).
 
-The `response.callback_args` and `error.callback_args` provide access to the arguments you passed via the `callback_args` option.
+## Usage
 
-> [!IMPORTANT]
-> Do not re-raise errors in the `on_error` callback as a means to retry the request. That will just retry the error callback job. If you want to retry the original request, you can enqueue a new request from within `on_error`. Be careful with this approach, though, as it can lead to infinite retry loops if the error condition is not resolved.
+### Make requests
 
-Callback jobs are not retried by default. If you want failed callbacks to be retried before being discarded, configure `retry_on` in an initializer:
+Use the `PatientHttp` module methods to make requests. There's a method for each HTTP method:
 
 ```ruby
-PatientHttp::SolidQueue::CallbackJob.retry_on StandardError, wait: :polynomially_longer, attempts: 5
+# GET request
+PatientHttp.get("https://api.example.com/users/123",
+  callback: MyCallback, callback_args: {user_id: 123})
+
+# POST request with a JSON body
+PatientHttp.post("https://api.example.com/users",
+  json: {name: "John", email: "john@example.com"},
+  callback: MyCallback)
+
+# PUT request
+PatientHttp.put("https://api.example.com/users/123",
+  json: {name: "Updated Name"},
+  callback: MyCallback)
+
+# PATCH request
+PatientHttp.patch("https://api.example.com/users/123",
+  json: {status: "active"},
+  callback: MyCallback)
+
+# DELETE request
+PatientHttp.delete("https://api.example.com/users/123",
+  callback: MyCallback)
 ```
 
-Note that Active Job runs `after_discard` hooks for any unhandled exception, not just when configured retries are exhausted. Without `retry_on`, the first callback failure will trigger the `on_retries_exhausted` handler and delete any externally stored payload, even though the failed job can still be retried manually from Mission Control (such retries will fail if the payload was stored externally).
->
-> Also note that the error callback is only called when an exception occurs during the HTTP request (timeout, connection failure, etc). HTTP error status codes (4xx, 5xx) do not trigger the error callback by default. Instead, they are treated as completed requests and passed to the `on_complete` callback. See the "Handling HTTP Error Responses" section below for how to treat HTTP errors as exceptions.
+The methods take these options:
+
+| Option | Description |
+| --- | --- |
+| `callback:` | Required. The callback service class, or its name. |
+| `callback_args:` | A Hash of arguments that the callback reads from the response or error. See [Callback arguments](#callback-arguments). |
+| `headers:` | The request headers. |
+| `body:` | The request body. GET, HEAD, and DELETE requests can't have a body. |
+| `json:` | An object to send as a JSON body. Can't be combined with `body:`. |
+| `params:` | Query parameters to add to the URL. |
+| `timeout:` | The request timeout in seconds. |
+| `raise_error_responses:` | Whether to treat non-2xx responses as errors. See [Handle HTTP error responses](#handle-http-error-responses). |
+| `processor:` | The name of the processor that runs the request. See [Named processors](#named-processors). |
+
+For all options, see the [patient_http documentation](https://github.com/bdurand/patient_http#make-requests).
+
+For more control, build a `PatientHttp::Request` object and pass it to `PatientHttp.execute`:
+
+```ruby
+request = PatientHttp::Request.new(:get, "https://api.example.com/users/123",
+  headers: {"Authorization" => "Bearer token"},
+  params: {include: "profile"},
+  timeout: 30
+)
+PatientHttp.execute(request: request, callback: MyCallback, callback_args: {user_id: 123})
+```
 
-### Handling HTTP Error Responses
+For the full `Request` and `Response` API reference, see the [patient_http documentation](https://github.com/bdurand/patient_http).
 
-By default, HTTP error status codes (4xx, 5xx) are treated as successful responses and passed to the `on_complete` callback. You can check the status using `response.success?`, `response.client_error?`, or `response.server_error?`:
+### Handle HTTP error responses
+
+By default, the gem treats HTTP error status codes (4xx and 5xx) as completed requests and passes them to `on_complete`. To check the status, use `response.success?`, `response.client_error?`, or `response.server_error?`:
 
 ```ruby
 class ApiCallback
@@ -132,22 +200,22 @@ PatientHttp.get(
 )
 ```
 
-If you prefer to treat HTTP errors as exceptions, you can use the `raise_error_responses` option. When enabled, non-2xx responses will call the `on_error` callback with an `HttpError` instead:
+To treat HTTP errors as exceptions, set the `raise_error_responses` option. With this option, a non-2xx response calls `on_error` with a `PatientHttp::HttpError` instead:
 
 ```ruby
 class ApiCallback
   def on_complete(response)
-    # Only called for 2xx responses
+    # Called only for 2xx responses.
     process_data(response.json)
   end
 
   def on_error(error)
-    # Called for exceptions AND HTTP errors when using raise_error_responses
+    # Called for exceptions, and for HTTP errors when raise_error_responses is set.
     if error.is_a?(PatientHttp::HttpError)
-      # Access the response via error.response
+      # The response is available from error.response.
       Rails.logger.error("HTTP #{error.status} from #{error.url}: #{error.response.body}")
     else
-      # Regular request errors (timeout, connection, etc)
+      # Request errors, such as timeouts and connection failures.
       Rails.logger.error("Request failed: #{error.message}")
     end
   end
@@ -160,7 +228,7 @@ PatientHttp.get(
 )
 ```
 
-The `HttpError` provides convenient access to the response:
+An `HttpError` gives you access to the request and the response:
 
 ```ruby
 def on_error(error)
@@ -170,97 +238,44 @@ def on_error(error)
     puts error.http_method         # HTTP method
     puts error.response.body       # Response body
     puts error.response.headers    # Response headers
-    puts error.response.json       # Parse JSON response (if applicable)
+    puts error.response.json       # Response body parsed as JSON
   end
 end
 ```
 
-## Usage Patterns
-
-### Making Requests
+### Named processors
 
-The primary interface for making requests is through the `PatientHttp` module, which provides convenience methods for all HTTP verbs:
+By default, all requests share one processor and one `max_connections` limit. If one process runs workloads with very different profiles, such as slow LLM API calls and fast webhook deliveries, a burst of one workload can use all the capacity that the other needs. Named processor profiles keep the workloads separate:
 
 ```ruby
-# GET request
-PatientHttp.get("https://api.example.com/users/123",
-  callback: MyCallback, callback_args: {user_id: 123})
-
-# POST request with JSON body
-PatientHttp.post("https://api.example.com/users",
-  json: {name: "John", email: "john@example.com"},
-  callback: MyCallback)
-
-# PUT request
-PatientHttp.put("https://api.example.com/users/123",
-  json: {name: "Updated Name"},
-  callback: MyCallback)
-
-# PATCH request
-PatientHttp.patch("https://api.example.com/users/123",
-  json: {status: "active"},
-  callback: MyCallback)
-
-# DELETE request
-PatientHttp.delete("https://api.example.com/users/123",
-  callback: MyCallback)
-```
-
-Available request options:
-
-- `callback:` - (required) Callback service class or class name
-- `callback_args:` - Hash of arguments passed to callback via response/error
-- `headers:` - Request headers
-- `body:` - Request body (for POST/PUT/PATCH)
-- `json:` - Object to serialize as JSON body (cannot use with body)
-- `params:` - Query parameters to append to URL
-- `timeout:` - Request timeout in seconds
-- `raise_error_responses:` - Treat non-2xx responses as errors
-
-You can also build a `PatientHttp::Request` object and pass it to `PatientHttp.execute` for more control:
-
-```ruby
-request = PatientHttp::Request.new(:get, "https://api.example.com/users/123",
-  headers: {"Authorization" => "Bearer token"},
-  params: {include: "profile"},
-  timeout: 30
-)
-PatientHttp.execute(request: request, callback: MyCallback, callback_args: {user_id: 123})
-```
-
-See the [patient_http docs](https://github.com/bdurand/patient_http) for the full `Request` and `Response` API reference.
-
-### Named Processors
-
-By default all requests share one processor and one `max_connections` cap. When one process serves workload classes with very different profiles (for example, large slow API calls and small fast webhook deliveries), a burst of one class can consume all of the capacity the other class needs. Named processor profiles isolate them:
-
-```ruby
-PatientHttp::SolidQueue.configure do |config|
+PatientHttp.configure do |config|
   config.processor(:llm, max_connections: 200, request_timeout: 120)
   config.processor(:webhooks, max_connections: 64, request_timeout: 10)
 end
 ```
 
-Each profile runs as an independent processor in the process, with its own capacity, timeouts, and threads. Profile options override the top-level configuration; anything not overridden (secrets, preprocessors, payload stores, encryption, logger) is shared. The `:default` processor always exists; declare `config.processor(:default, ...)` to override its options.
+Each profile runs as an independent processor in the process, with its own capacity, timeouts, and threads. Profile options override the top-level configuration. The profiles share every option that they don't override, such as secrets, preprocessors, payload stores, encryption, and the logger. The `:default` processor always exists. To override its options, declare `config.processor(:default, ...)`.
 
-Route a request to a processor in any of these ways:
+To send a request to a processor, use any of these methods:
 
 ```ruby
-# Explicit option on execute
-PatientHttp::SolidQueue.execute(request, callback: MyCallback, processor: :llm)
+# An option on the request method.
+PatientHttp.get(url, callback: MyCallback, processor: :llm)
 
-# On the request itself (survives serialization, retries, and crash recovery)
+# A request object. The processor is kept through serialization, retries, and crash recovery.
 request = PatientHttp::Request.new(:get, url, processor: :llm)
 
-# Through a request template
+# A request template.
 template = PatientHttp::RequestTemplate.new(base_url: url, processor: :llm)
 ```
 
-The processor name is serialized into the job arguments, so Active Job retries and crash recovery keep their routing. A job that names a processor that is not configured in the executing process raises `PatientHttp::UnknownProcessorError` and is retried with backoff; this makes new profile names safe to roll out gradually. Jobs enqueued by older gem versions run on the `:default` processor.
+A request that names a processor that isn't declared in the process making the request raises `PatientHttp::UnknownProcessorError`, so a misspelled name fails where the request is made. Declare processors in an initializer that every process loads, such as the web server and the Solid Queue workers, not only in code that runs in worker processes.
 
-### Using Request Templates
+The processor name is saved in the job arguments, so Active Job retries and crash recovery send the request to the same processor. If a job names a processor that isn't configured in the process that runs it, the job raises `PatientHttp::UnknownProcessorError`, and Active Job retries it with backoff. As a result, you can roll out a new profile name gradually. Jobs enqueued by earlier versions of the gem run on the `:default` processor.
 
-For repeated requests to the same API, use `PatientHttp::RequestTemplate` to share configuration:
+### Use request templates
+
+To share settings across requests to the same API, use `PatientHttp::RequestTemplate`:
 
 ```ruby
 class ApiService
@@ -292,11 +307,11 @@ class ApiService
 end
 ```
 
-### Using the RequestHelper Module
+If the template doesn't set a `timeout`, the configured `request_timeout` applies.
 
-For classes that make many async HTTP requests, you can include `PatientHttp::RequestHelper` to get convenient instance methods like `async_get`, `async_post`, `async_put`, `async_patch`, and `async_delete`. You can also define a request template at the class level using the `request_template` class method to set shared options like `base_url`, `headers`, and `timeout`.
+### Use the RequestHelper module
 
-When using this gem, the request handler is automatically registered when you call `PatientHttp::SolidQueue.configure` or when the processor starts — no manual setup is required. The handler stays registered when the processor stops, so a request submitted while the worker is shutting down is enqueued as a job and executed by the next process instead of being lost.
+For a class that makes many requests, include `PatientHttp::RequestHelper`. The module adds the `async_get`, `async_head`, `async_post`, `async_put`, `async_patch`, `async_delete`, `async_query`, and `async_request` instance methods. To set shared options such as `base_url`, `headers`, and `timeout`, use the `request_template` class method:
 
 ```ruby
 class NotificationService
@@ -323,18 +338,18 @@ class NotificationService
 end
 ```
 
-The `async_*` methods accept the same options as `PatientHttp.get`, `PatientHttp.post`, etc. Paths are resolved relative to the `base_url` defined in the request template.
+The `async_*` methods take the same options as `PatientHttp.get`, `PatientHttp.post`, and the other module methods. Paths are relative to the template's `base_url`.
 
-See the [patient_http gem](https://github.com/bdurand/patient_http) for the full `RequestHelper` documentation.
+For the full `RequestHelper` documentation, see the [patient_http documentation](https://github.com/bdurand/patient_http#use-the-requesthelper-module).
 
-### Callback Arguments
+### Callback arguments
 
-Pass custom data to your callbacks using the `callback_args` option:
+To pass data to your callbacks, use the `callback_args` option:
 
 ```ruby
 class FetchDataCallback
   def on_complete(response)
-    # Access callback_args using symbol or string keys
+    # Read callback_args with symbol or string keys.
     user_id = response.callback_args[:user_id]
     request_timestamp = response.callback_args[:request_timestamp]
 
@@ -354,7 +369,7 @@ class FetchDataCallback
   end
 end
 
-# Pass data via callback_args option
+# Pass data with the callback_args option.
 PatientHttp.get(
   "https://api.example.com/users/#{user_id}",
   callback: FetchDataCallback,
@@ -365,137 +380,185 @@ PatientHttp.get(
 )
 ```
 
-**Important details about callback_args:**
+The `callback_args` value follows these rules:
+
+- It must be a Hash, or respond to `to_h`, and contain only JSON-native types: `nil`, `true`, `false`, `String`, `Integer`, `Float`, `Array`, and `Hash`.
+- Hash keys are converted to strings, including the keys of nested hashes and of hashes in arrays.
+- You can read the arguments with symbol or string keys: `callback_args[:user_id]` or `callback_args["user_id"]`.
+- Reading a key that isn't set raises a `KeyError`. To get a default value instead, use `callback_args.fetch(:user_id, nil)`.
+
+### Protect sensitive data
 
-- Must be a Hash (or respond to `to_h`) containing only JSON-native types: `nil`, `true`, `false`, `String`, `Integer`, `Float`, `Array`, or `Hash`
-- Hash keys will be converted to strings for serialization
-- Nested hashes and hashes in arrays also have their keys converted to strings
-- You can access callback_args using either symbol or string keys: `callback_args[:user_id]` or `callback_args["user_id"]`
+The gem stores requests and responses in your queue database so that it can run the callback job. If they contain sensitive data, that data is stored in plain text.
 
-### Sensitive Data Handling
+To protect the data, configure encryption. The gem then encrypts all request and response data before it stores the data in the queue, and decrypts the data when it reads it.
 
-Requests and responses from asynchronous HTTP requests may be stored in your queue backend (and optionally external storage) in order to execute completion callbacks. This can raise security concerns if they contain sensitive data since the data will be stored in plain text.
+#### Use an encryption key
 
-Encryption is configured on the parent `patient_http` gem. You can set an `encryption_key` to automatically encrypt and decrypt request and response data using `ActiveSupport::MessageEncryptor`:
+The simplest option is `encryption_key=`. It uses [ActiveSupport::MessageEncryptor](https://api.rubyonrails.org/classes/ActiveSupport/MessageEncryptor.html) with AES-256-GCM:
 
 ```ruby
-PatientHttp::SolidQueue.configure do |config|
-  config.encryption_key = Rails.application.credentials.patient_http_secret
+PatientHttp.configure do |config|
+  config.encryption_key = Rails.application.credentials.patient_http_encryption_key
 end
 ```
 
-See the [patient_http gem](https://github.com/bdurand/patient_http) for full documentation on encryption options, including key rotation and custom encryption callables.
+To rotate keys, pass an array. The first key encrypts data, and all keys are tried for decryption:
+
+```ruby
+PatientHttp.configure do |config|
+  config.encryption_key = [
+    Rails.application.credentials.patient_http_encryption_key,
+    Rails.application.credentials.patient_http_old_key
+  ]
+end
+```
+
+#### Use custom callables
+
+To use another encryption library, provide callables that take and return raw bytes as a String:
+
+```ruby
+PatientHttp.configure do |config|
+  config.encryption { |bytes| MyEncryption.encrypt(bytes) }
+  config.decryption { |bytes| MyEncryption.decrypt(bytes) }
+end
+```
+
+You can also pass any object that responds to `call`:
+
+```ruby
+PatientHttp.configure do |config|
+  config.encryption(->(bytes) { MyEncryption.encrypt(bytes) })
+  config.decryption(->(bytes) { MyEncryption.decrypt(bytes) })
+end
+```
+
+To keep API tokens out of the queue entirely, use secrets instead. For secrets, request preprocessors, and payload stores for large payloads, see the [patient_http documentation](https://github.com/bdurand/patient_http#sensitive-and-large-payloads).
 
 ## Configuration
 
-The gem can be configured globally in an initializer:
+All configuration is optional. To set options, call `PatientHttp.configure` in an initializer. The method yields this gem's configuration. `PatientHttp::SolidQueue.configure` does the same thing, but `PatientHttp.configure` keeps the initializer free of references to the job system.
+
+Every call yields the same configuration object, so options accumulate. Several initializers can each set options without overwriting one another.
 
 ```ruby
-PatientHttp::SolidQueue.configure do |config|
-  # Maximum concurrent HTTP requests (default: 256)
+PatientHttp.configure do |config|
+  # Maximum concurrent HTTP requests (default: 256).
   config.max_connections = 256
 
-  # Default timeout for HTTP requests in seconds (default: 60)
+  # Default timeout for HTTP requests in seconds (default: 60).
   config.request_timeout = 60
 
-  # Maximum number of host clients to pool (default: 100)
+  # Maximum number of host clients to pool (default: 100).
   config.connection_pool_size = 100
 
-  # Connection timeout in seconds (default: nil, uses request_timeout)
+  # Timeout in seconds to open a connection, including the TCP connect and the
+  # TLS handshake (default: nil, no limit). It doesn't limit the wait for a
+  # response; request_timeout does that.
   config.connection_timeout = 10
 
-  # Number of retries for failed requests (default: 3)
-  config.retries = 3
+  # TCP keepalive for pooled connections (default: nil, the kernel sends no
+  # probes). A number sets the idle seconds before the first probe. A Hash also
+  # sets the interval and the probe count, for example
+  # {idle: 30, interval: 10, count: 3}. The Hash must contain :idle. The
+  # :interval default is 10 seconds, and the :count default is 3 probes.
+  config.tcp_keepalive = 30
 
-  # Handler called when a callback job exhausts all Sidekiq retries
-  config.on_retries_exhausted { |error| MyAlertService.notify(error) }
+  # Seconds that sent data can stay unacknowledged before the kernel closes the
+  # connection (default: nil, the kernel default applies). Sets
+  # TCP_USER_TIMEOUT, which is available only on Linux.
+  config.tcp_user_timeout = 30
+
+  # Number of retries for failed requests (default: 3).
+  config.retries = 3
 
-  # HTTP/HTTPS proxy URL (default: nil)
-  # Supports authentication: "http://user:pass@proxy.example.com:8080"
+  # HTTP or HTTPS proxy URL (default: nil). Supports authentication, for
+  # example "http://user:pass@proxy.example.com:8080".
   config.proxy_url = "http://proxy.example.com:8080"
 
-  # Default User-Agent header for all requests (default: "PatientHttp")
+  # Default User-Agent header for all requests (default: "PatientHttp").
   config.user_agent = "MyApp/1.0"
 
-  # Timeout for graceful shutdown in seconds
-  # (default: SolidQueue.shutdown_timeout - 2)
-  # This should be less than your worker shutdown timeout
+  # Timeout for graceful shutdown in seconds (default: the Solid Queue shutdown
+  # timeout minus 2 seconds). Must be less than Solid Queue's shutdown timeout.
   config.shutdown_timeout = 23
 
-  # Maximum response body size in bytes (default: 1MB)
-  # Responses larger than this will trigger ResponseTooLargeError
+  # Maximum response body size in bytes (default: 1MB). Larger responses raise
+  # ResponseTooLargeError.
   config.max_response_size = 1024 * 1024
 
-  # Maximum number of redirects to follow (default: 5, 0 disables)
+  # Maximum number of redirects to follow (default: 5; 0 turns off redirects).
   config.max_redirects = 5
 
-  # Whether to raise HttpError for non-2xx responses by default (default: false)
+  # Whether to raise HttpError for non-2xx responses by default (default: false).
   config.raise_error_responses = false
 
-  # Heartbeat interval for crash recovery in seconds (default: 60)
+  # Heartbeat interval for crash recovery in seconds (default: 60).
   config.heartbeat_interval = 60
 
-  # Orphan detection threshold in seconds (default: 300)
-  # Requests older than this without a heartbeat will be re-enqueued
+  # Seconds without a heartbeat after which a request is re-enqueued
+  # (default: 300).
   config.orphan_threshold = 300
 
-  # Size threshold in bytes for external payload storage (default: 64KB)
-  # Payloads larger than this will be stored externally when a payload
-  # store is configured.
+  # Size in bytes above which payloads are stored externally when a payload
+  # store is configured (default: 64KB).
   config.payload_store_threshold = 64 * 1024
 
-  # Queue name for RequestJob and CallbackJob (default: nil, Active Job default)
-  config.queue_name = "async_http"
+  # Queue name for RequestJob and CallbackJob (default: nil, the Active Job
+  # default queue).
+  config.queue_name = "patient_http"
 
-  # Number of threads that decode responses and deliver results (default: 2)
+  # Number of threads that decode responses and deliver results (default: 2).
   config.completion_threads = 2
 
-  # Maximum connections per host (default: nil, unlimited)
+  # Maximum connections to each host (default: nil, no limit).
   config.max_connections_per_host = 32
 
-  # Named processor profiles for workload isolation (see Named Processors)
+  # Named processor profiles for workload isolation. See Named processors.
   config.processor(:llm, max_connections: 200, request_timeout: 120)
   config.processor(:webhooks, max_connections: 64, request_timeout: 10)
 
-  # Custom logger (defaults to SolidQueue.logger)
+  # Handler that runs when a callback job is discarded. See Handle exhausted
+  # retries.
+  config.on_retries_exhausted { |error| MyAlertService.notify(error) }
+
+  # Logger (default: SolidQueue.logger).
   config.logger = Rails.logger
 
-  # Encryption key for sensitive data (see Sensitive Data Handling)
-  # Accepts a string or an array of strings for key rotation.
-  # Encryption is provided by the parent patient_http gem.
-  config.encryption_key = Rails.application.credentials.patient_http_secret
+  # Encryption for sensitive data. See Protect sensitive data.
+  config.encryption_key = Rails.application.credentials.patient_http_encryption_key
 end
 ```
 
-See the [Configuration](lib/patient_http/solid_queue/configuration.rb) class for all available options.
+For all options, see the [Configuration](lib/patient_http/solid_queue/configuration.rb) class. For the HTTP options that this gem inherits, see the [patient_http documentation](https://github.com/bdurand/patient_http#configuration).
 
-### Tuning Tips
+### Tuning tips
 
-- `max_connections`: Adjust this based on your system's resources. Each connection uses memory and file descriptors. A tuned system with sufficient resources can handle thousands of concurrent connections.
-- `request_timeout`: Set this based on the expected response times of the APIs you are calling. AI APIs might sometimes take minutes to respond as they generate content.
-- `connection_pool_size`: Controls how many connections to different hosts are kept alive. Increase for applications calling many different API endpoints.
-- `connection_timeout`: Set this if you need to fail fast on connection establishment. Useful for detecting network issues quickly.
-- `retries`: Number of times to retry a failed request before calling the error callback.
-- `max_response_size`: Set this to limit the maximum size of HTTP responses. This helps prevent excessive memory usage from unexpectedly large responses. Responses need to be serialized as Active Job arguments and very large responses may cause performance issues. If a response body is text content, it will be compressed to save space. However, binary content needs to be Base64 encoded which increases size by ~33%.
-- `payload_store_threshold`: Lower this if your queue backend struggles with large payloads; higher values avoid extra external storage reads/writes.
-- `max_connections_per_host`: Bounds sockets per host. Verify the process file descriptor limit covers `max_connections` plus pooled idle host connections plus the application's own connections; raise the limit if needed.
-- `completion_threads`: Number of threads that decode responses and deliver results (default 2). Increase when result callbacks do heavier work and completions back up behind them. Size the Active Record connection pool to cover these threads plus the task monitor thread in addition to the worker threads.
-- `shutdown_timeout`: Must be below the process supervisor's termination window so the drain finishes before a hard kill. The default derives it from Solid Queue's own shutdown timeout; check any additional supervisor stop timeout as well.
-- `heartbeat_interval` and `orphan_threshold`: For high-churn workloads, keep `heartbeat_interval` as large as your recovery SLO allows (while still less than `orphan_threshold`) to reduce write/update pressure on monitoring tables. If Solid Queue uses PostgreSQL and request volume is high, tune autovacuum for the queue database tables because `inflight_requests` is intentionally insert/update/delete heavy.
+- `max_connections`: Set this based on your system's resources. Each connection uses memory and a file descriptor. A tuned system with enough resources can handle thousands of concurrent connections.
+- `request_timeout`: Set this based on the response times of the APIs that you call. AI APIs can take minutes to respond while they generate content.
+- `connection_pool_size`: Sets the maximum number of hosts whose connections are kept open. Increase it if your application calls many different hosts.
+- `connection_timeout`: Limits only the TCP connect and the TLS handshake. Set it to fail fast when a host doesn't answer. It doesn't limit the wait for a response, because `request_timeout` controls the full exchange.
+- `retries`: Sets the number of times to retry a failed request before the gem calls the error callback.
+- `max_response_size`: Limits the size of HTTP responses to prevent high memory use from unexpectedly large responses. Responses are serialized in Active Job arguments, and very large responses can slow the queue database down. Text response bodies are compressed to save space. Binary bodies are Base64 encoded, which increases their size by about 33%.
+- `payload_store_threshold`: Lower this if large payloads slow your queue down. Higher values avoid extra reads and writes to the payload store.
+- `max_connections_per_host`: Limits the sockets open to each host. Make sure that the process file descriptor limit covers `max_connections`, plus idle pooled connections, plus the application's own connections. Raise the limit if needed.
+- `shutdown_timeout`: Must be less than the process supervisor's stop timeout, so that in-flight requests finish before a hard kill. The default is based on Solid Queue's shutdown timeout. If a container orchestrator or init system also stops the process, check its stop timeout as well.
+- `completion_threads`: Increase this when result delivery does heavy work, such as serialization or encryption, and finished requests wait for a thread. Size the Active Record connection pool to cover these threads, the task monitor thread, and the worker threads.
+- `heartbeat_interval` and `orphan_threshold`: For high-volume workloads, set `heartbeat_interval` as high as your recovery objective allows, while you keep it less than `orphan_threshold`. Fewer heartbeats mean fewer writes to the crash-recovery tables. If Solid Queue uses PostgreSQL and the request volume is high, tune autovacuum for the queue database tables. The `inflight_requests` table has many inserts, updates, and deletes by design.
 
 > [!IMPORTANT]
+> When the processor reaches `max_connections`, a new request raises an error in its Active Job. The job retries with polynomial backoff until the processor has capacity.
 >
-> One difference between using this gem and making synchronous HTTP requests from a Solid Queue job is that if `max_connections` is reached due to slow asynchronous requests, new requests will trigger an error on the Active Job. The job declares `retry_on` for this error with a polynomial backoff, so it will automatically be retried until the processor has capacity again.
->
-> In contrast, slow synchronous HTTP requests will fill up the worker pool and block new jobs from being dequeued until a worker thread becomes free.
+> Synchronous HTTP requests in Solid Queue jobs behave differently. Slow synchronous requests fill the worker pool, and no new jobs start until a worker thread is free.
 >
-> In general, the former behavior is preferable because it allows Solid Queue to continue processing other jobs and prevents getting into a state with 1000's of jobs stuck in the queue.
+> The asynchronous behavior is usually better, because Solid Queue keeps running other jobs, and thousands of jobs don't pile up in the queue.
 
-## Metrics and Monitoring
+## Metrics and monitoring
 
-### Callbacks for Custom Monitoring
+### Monitoring callbacks
 
-You can register callbacks to integrate with your monitoring system using the `after_completion` and `after_error` hooks:
+To send metrics to your monitoring system, register `after_completion` and `after_error` callbacks:
 
 ```ruby
 PatientHttp::SolidQueue.after_completion do |response|
@@ -504,86 +567,101 @@ PatientHttp::SolidQueue.after_completion do |response|
 end
 
 PatientHttp::SolidQueue.after_error do |error|
-  error_type = error.is_a?(PatientHttp::Error) ? error.error_type : "exception"
-  StatsD.increment("patient_http.error.#{error_type}")
-  Rails.logger.error("Async HTTP error: #{error.class.name} - #{error.message}")
+  StatsD.increment("patient_http.error.#{error.error_type}")
+  Sentry.capture_message("Async HTTP error: #{error.message}")
+end
+```
+
+You can register more than one callback. Callbacks run in the order that you register them.
+
+### Handle exhausted retries
+
+Callback jobs aren't retried by default. To retry a failed callback job before Active Job discards it, configure `retry_on` in an initializer:
+
+```ruby
+PatientHttp::SolidQueue::CallbackJob.retry_on StandardError, wait: :polynomially_longer, attempts: 5
+```
+
+When Active Job discards a callback job, the gem can call an `on_retries_exhausted` handler. Use the handler to send an alert or to record that a callback failed permanently. The handler receives the same error object as `on_error`:
+
+```ruby
+PatientHttp.configure do |config|
+  config.on_retries_exhausted do |error|
+    Sentry.capture_message("Callback permanently failed: #{error.message}")
+    DeadLetterRecord.create!(
+      error_message: error.message,
+      callback_args: error.callback_args
+    )
+  end
+end
+```
+
+You can also assign any object that responds to `call`:
+
+```ruby
+PatientHttp.configure do |config|
+  config.on_retries_exhausted = ->(error) { MyAlertService.notify(error) }
 end
 ```
 
-You can register multiple callbacks; they will be called in the order registered.
+> [!NOTE]
+> The gem calls the `on_retries_exhausted` handler only for callback jobs that deliver an error to `on_error`. If the handler raises an exception, the gem logs a warning, and the discarded job cleanup continues as usual.
+>
+> Active Job runs `after_discard` hooks for any unhandled exception, not only when the configured retries run out. Without `retry_on`, the first callback failure calls the `on_retries_exhausted` handler and deletes any externally stored payload. You can still retry the failed job by hand from Mission Control, but the retry fails if the payload was stored externally.
 
-## Shutdown Behavior
+## Shutdown behavior
 
-The async HTTP processor automatically hooks in with Solid Queue's lifecycle events.
+The async HTTP processor follows Solid Queue's lifecycle events:
 
-1. **Startup:** Processor starts automatically when Solid Queue starts a worker
-2. **Shutdown:** Processor waits up to `shutdown_timeout` seconds for in-flight requests to complete
+1. **Startup**: The processor starts when Solid Queue starts a worker.
+2. **Shutdown**: The processor waits up to `shutdown_timeout` seconds for in-flight requests to finish.
 
-### Incomplete Request Handling
+### Incomplete requests
 
-If requests are still in-flight when shutdown times out:
+If requests are still in flight when the shutdown timeout ends, the gem interrupts them and re-enqueues their Active Jobs. The jobs run again when Solid Queue restarts or on another worker, so no work is lost during deployments or restarts.
 
-- In-flight requests are interrupted
-- The **original Active Job** is automatically re-enqueued
-- Re-enqueued jobs will be processed again when workers are available
+### Crash recovery
 
-This ensures no work is lost during deployments or restarts.
+The gem recovers requests from processes that crash:
 
-### Crash Recovery
+1. **Heartbeats**: Every `heartbeat_interval` seconds, each process updates the heartbeat times of its in-flight requests in the database.
+2. **Orphan detection**: One process at a time checks for requests that haven't had a heartbeat in `orphan_threshold` seconds.
+3. **Re-enqueue**: The gem re-enqueues the Active Jobs of the orphaned requests.
 
-The gem includes crash recovery to handle process failures:
+As a result, if a Solid Queue worker process crashes, another process retries its in-flight requests.
 
-1. **Heartbeat Tracking:** Every `heartbeat_interval` seconds, the processor updates heartbeat timestamps for all in-flight requests in the database
-2. **Orphan Detection:** One processor periodically checks for requests that haven't received a heartbeat update in `orphan_threshold` seconds
-3. **Automatic Re-enqueue:** Orphaned requests have their original Active Jobs re-enqueued
+Crash recovery gives at-least-once delivery. If a process crashes at the wrong moment, such as between a re-enqueue and the removal of the registry entry, a request can run more than once, and its callback can run more than once. Make your callbacks idempotent. If the gem can't write a request's registry entry, the processor rejects the request, the job raises `PatientHttp::SolidQueue::RegistrationError`, and Active Job retries it.
 
-This ensures that if a worker process crashes, its in-flight requests will be retried by another process.
+If the gem can't hand a result to a callback job, it keeps the request's registry entry, so crash recovery re-enqueues the request.
 
 ## Testing
 
-The gem supports testing with Active Job test adapters. When in test mode (`PatientHttp.testing?`), async HTTP requests are executed immediately within the worker thread, blocking until completion. This allows you to write tests that verify the full request/response cycle without needing the async processor to be running.
+The gem supports the Active Job test adapters. When `RAILS_ENV`, `RACK_ENV`, or `APP_ENV` is `test`, requests run immediately in the worker thread and block until they finish. As a result, tests can check the full request and response cycle without a running processor.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Add the gem to your Gemfile:
 
 ```ruby
 gem "patient_http-solid_queue"
 ```
 
-Then execute:
+Then install it:
 
 ```bash
 bundle install
 ```
 
-Install and run the gem migrations:
+Run the install generator, and then run the migration:
 
 ```bash
-bin/rails patient_http_solid_queue:install:migrations
+bin/rails generate patient_http:solid_queue:install
 bin/rails db:migrate
 ```
 
-The database tables are used for crash recovery and monitoring of in-flight requests and need to be added to the same database that Solid Queue uses.
-
-By default, this install task copies migrations to the `queue` database migration path (typically `db/queue_migrate`).
-If your Solid Queue database name is different, override it with `DATABASE=your_database_name`.
+The generator creates a migration for the crash-recovery and in-flight request tables. It also creates a commented initializer, which you can edit or delete.
 
-For a typical multi-database setup, ensure your `queue` database config defines its own migration path:
-
-```yaml
-development:
-  primary:
-    adapter: sqlite3
-    database: storage/development.sqlite3
-  queue:
-    adapter: sqlite3
-    database: storage/development_queue.sqlite3
-    migrations_paths:
-      - db/queue_migrate
-```
-
-PostgreSQL example:
+The tables must be in the database that Solid Queue uses. The generator finds that database from `config.solid_queue.connects_to`, or from the database names in `config/database.yml`, and writes the migration to its migrations path, so a multi-database application needs no extra arguments. The generator prints the migrate command for the database that it chose. For a typical setup like the following, the command is `bin/rails db:migrate:queue`:
 
 ```yaml
 development:
@@ -597,51 +675,42 @@ development:
       - db/queue_migrate
 ```
 
-Then run:
+If the generator can't find the database, name it:
 
 ```bash
-bin/rails patient_http_solid_queue:install:migrations
-bin/rails db:queue:migrate
+bin/rails generate patient_http:solid_queue:install --database=solid_queue
 ```
 
-If your Solid Queue database is not named `queue`, pass its name explicitly when installing migrations:
-
-```bash
-bin/rails patient_http_solid_queue:install:migrations DATABASE=solid_queue
-bin/rails db:migrate:solid_queue
-```
+To generate only the migration, pass `--skip-initializer`.
 
 ## Contributing
 
 Open a pull request on [GitHub](https://github.com/bdurand/patient_http-solid_queue).
 
-Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
+Follow the [standardrb](https://github.com/testdouble/standard) style, and run `standardrb --fix` before you submit a pull request.
 
-Run the test suite with:
+Run the tests:
 
 ```bash
 bundle exec rake
 ```
 
-There is also a bundled test app in the `test_app` directory that can be used for manual testing and experimentation.
-
-To run the test app, first install the dependencies:
+The `test_app` directory has a test app for manual testing. To run it, install its dependencies:
 
 ```bash
 bundle exec rake test_app:bundle
 ```
 
-The server will run on http://localhost:9292 and can be started with:
+Then start the server, which runs at http://localhost:9292:
 
 ```bash
 bundle exec rake test_app
 ```
 
-## Further Reading
+## Further reading
 
 - [Architecture](ARCHITECTURE.md)
 
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).