patient_http 1.6.0 → 1.7.0

data/README.md

@@ -6,230 +6,222 @@
 
 *Built for APIs that like to think.*
 
-Generic async HTTP connection pool for Ruby applications using Fiber-based concurrency.
+This gem runs HTTP requests on a dedicated async I/O processor and passes each result to a callback service. Application threads don't wait for HTTP responses, so they're free to do other work while requests are in flight.
 
 ## Motivation
 
-Applications that make HTTP requests from within threaded environments often find that threads block waiting for I/O. A single slow API response holds an entire thread hostage, preventing it from doing other work. When many threads are blocked on HTTP I/O simultaneously, throughput collapses.
+In a threaded application, a thread that makes an HTTP request waits for the response. A slow API response holds the thread for the full duration, so the thread can't do other work. When many threads wait on HTTP requests at the same time, throughput drops.
 
-PatientHttp solves this by running HTTP requests in a dedicated processor thread that uses Ruby's Fiber scheduler for non-blocking I/O. Application threads hand off HTTP requests to the processor and return immediately. The processor handles hundreds of concurrent HTTP connections using fibers, then notifies the application when responses arrive via a pluggable callback mechanism.
+This gem runs HTTP requests in a dedicated processor thread that uses Ruby's fiber scheduler for non-blocking I/O. An application thread hands off a request to the processor and returns immediately. The processor runs hundreds of HTTP requests at the same time. When a response arrives, the processor passes it to your callback service through a job system.
 
-This design keeps application threads free to do other work while HTTP requests are in flight.
+## Quick start
 
-In general you will want to use this gem through an integration like [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) or [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue). These gems provide a request handler that integrates with their respective job processing systems, allowing you to enqueue HTTP requests directly from your application code without coupling it to the underlying processor implementation. See the [integration](#integration) section for details.
+The processor needs a process to run in and a job system to deliver the results. Install the integration gem for your job system, which handles both:
 
-The [patient_llm](https://github.com/bdurand/patient_llm) gem provides an integration for making large language model requests asynchronously. This was the original motivation for building PatientHttp because LLM requests can take much longer than typical HTTP requests.
+- [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) for Sidekiq
+- [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) for Solid Queue
 
-## Quick Start
+For large language model (LLM) requests, use [patient_llm](https://github.com/bdurand/patient_llm), which builds on this gem. LLM requests can take much longer than typical HTTP requests, which was the original reason for this gem.
 
-### 1. Implement a TaskHandler
+### 1. Install the gem
 
-The `TaskHandler` is the integration point between the pool and your application. It defines what happens when a request completes, fails, or needs to be retried.
+Add the integration gem for your job system to your Gemfile. The integration gem depends on this gem, so you don't need to add `patient_http` too.
 
 ```ruby
-class MyTaskHandler < PatientHttp::TaskHandler
-  def initialize(job_id)
-    @job_id = job_id
-  end
+gem "patient_http-sidekiq"
+```
 
-  def on_complete(response, callback)
-    # Enqueue a message for your application to process the response.
-    # Keep this lightweight and thread-safe -- it runs on a completion
-    # worker thread, concurrently with other completions.
-    MyJobSystem.enqueue(callback, :on_complete, response.as_json)
-  end
+No other setup is required. When the integration gem loads, it registers the request handler and connects the processor to the startup and shutdown of your job system. You don't need to write an initializer or call a setup method.
 
-  def on_error(error, callback)
-    MyJobSystem.enqueue(callback, :on_error, error.as_json)
+### 2. Create a callback service
+
+Define a callback service class with `on_complete` and `on_error` instance methods. The callback service runs in a background job, so it can do slow work.
+
+```ruby
+class FetchUserCallback
+  def on_complete(response)
+    user_id = response.callback_args[:user_id]
+    User.find(user_id).update!(external_data: response.json)
   end
 
-  def retry
-    # Re-enqueue the original job for retry when the processor
-    # shuts down with in-flight requests
-    MyJobSystem.enqueue_job(@job_id)
+  def on_error(error)
+    user_id = error.callback_args[:user_id]
+    Rails.logger.error("Failed to fetch user #{user_id}: #{error.message}")
   end
 end
 ```
 
-> **Important:** TaskHandler callbacks run on the processor's completion worker threads (see `completion_threads`), not the reactor thread, so they no longer block the event loop. Keep them lightweight anyway -- typically just enqueuing a message for another system to pick up. Heavy callbacks compete with the reactor for the GVL, and because a task stays in the capacity count until its result is delivered, callbacks that back up consume request capacity.
->
-> Callbacks must be thread-safe. Results are delivered concurrently on `completion_threads` workers (default 2), so two callbacks can run at the same time and in an order unrelated to the order the requests completed. Set `completion_threads: 1` to serialize delivery.
->
-> Callbacks must also be idempotent. A callback that raises is retried `completion_retries` times (default 2), so one that raises after enqueuing its message enqueues it again. Set `completion_retries: 0` if that is not acceptable.
-
-### 2. Create and Enqueue Requests
+### 3. Make HTTP requests
 
 ```ruby
-# Configure the processor
-config = PatientHttp::Configuration.new(
-  max_connections: 256,
-  request_timeout: 60
+PatientHttp.get(
+  "https://api.example.com/users/123",
+  callback: FetchUserCallback,
+  callback_args: {user_id: 123}
 )
+```
 
-# Start the processor
-processor = PatientHttp::Processor.new(config)
-processor.start
+The call returns immediately. The processor runs the request, and when the request finishes, a background job calls your callback's `on_complete` method. If the request fails, the job calls `on_error` instead.
 
-# Build a request
-request = PatientHttp::Request.new(
-  :get,
-  "https://api.example.com/users/123",
-  headers: {"Authorization" => "Bearer token"}
-)
+Every option has a working default. To change the defaults, see [Configuration](#configuration).
 
-# Create a task with your handler
-task = PatientHttp::RequestTask.new(
-  request: request,
-  task_handler: MyTaskHandler.new("job-123"),
-  callback: "FetchDataCallback",
-  callback_args: {user_id: 123}
-)
+### Run requests without a job system
 
-# Enqueue it
-processor.enqueue(task)
+In consoles, tests, and development environments without a job system, run requests inline:
+
+```ruby
+PatientHttp.inline!
 ```
 
-### 3. Process Callbacks
+After this call, each request made with the `PatientHttp` module methods runs immediately on the calling thread. The request goes through the full request lifecycle, including timeouts, redirects, and error handling, and the callback runs before the request method returns.
 
-When the HTTP request completes, your `TaskHandler#on_complete` is called with the `Response` and callback class name. Your handler is responsible for invoking the callback in whatever way makes sense for your application (e.g., enqueuing a background job).
+```ruby
+PatientHttp.inline!
+PatientHttp.get("https://api.example.com/users/123", callback: FetchUserCallback)
+# FetchUserCallback#on_complete has already run.
+```
+
+Requests that callbacks make also run inline. To check whether the inline handler is registered, call `PatientHttp.inline?`. To run one request inline without registering a handler, call `PatientHttp.execute_inline(request:, callback:)`.
+
+## Usage
+
+### Make requests
+
+The `PatientHttp` module has a method for each HTTP method: `get`, `head`, `post`, `put`, `patch`, `delete`, and `query`. The methods take the same options. `PatientHttp.request` takes the HTTP method as its first argument.
 
 ```ruby
-class FetchDataCallback
-  def on_complete(response)
-    user_id = response.callback_args[:user_id]
-    data = response.json
-    User.find(user_id).update!(external_data: data)
-  end
+PatientHttp.post(
+  "https://api.example.com/users",
+  json: {name: "John", email: "john@example.com"},
+  callback: CreateUserCallback,
+  callback_args: {source: "signup"}
+)
+```
 
-  def on_error(error)
-    user_id = error.callback_args[:user_id]
-    Rails.logger.error("Failed for user #{user_id}: #{error.message}")
-  end
-end
+The methods take these options:
+
+| Option | Description |
+| --- | --- |
+| `callback:` | Required. The callback service class, or its name. |
+| `callback_args:` | A Hash of arguments that the callback reads from the response or error. See [Callback arguments](#callback-arguments). |
+| `headers:` | The request headers. |
+| `body:` | The request body. GET, HEAD, and DELETE requests can't have a body. |
+| `json:` | An object to send as a JSON body. Can't be combined with `body:`. |
+| `params:` | Query parameters to add to the URL. |
+| `timeout:` | The request timeout in seconds. |
+| `raise_error_responses:` | Whether to treat non-2xx responses as errors. See [Handle HTTP error responses](#handle-http-error-responses). |
+| `max_redirects:` | The maximum number of redirects to follow for this request. `0` turns off redirects. |
+| `follow_method_changing_redirects:` | Whether to follow redirects that change the HTTP method. See [Redirects](#redirects). |
+| `redirect_strip_headers:` | The names of headers to remove from redirected requests. |
+| `preprocessors:` | The names of registered [preprocessors](#request-preprocessors) to run on the request. |
+| `processor:` | The name of the processor that runs the request. See [Named processors](#named-processors). |
+
+For more control, build a `PatientHttp::Request` object and pass it to `PatientHttp.execute`:
+
+```ruby
+request = PatientHttp::Request.new(:get, "https://api.example.com/users/123",
+  headers: {"Authorization" => PatientHttp.secret(:api_token)},
+  params: {include: "profile"},
+  timeout: 30
+)
+PatientHttp.execute(request: request, callback: FetchUserCallback, callback_args: {user_id: 123})
 ```
 
-## Handling HTTP Error Responses
+Application code that uses these methods never names the job system. To move to another job system, change the integration gem in your Gemfile.
 
-By default, HTTP error status codes (4xx, 5xx) are treated as completed requests. You can check the status using helper methods on the response:
+### Handle HTTP error responses
+
+By default, HTTP error status codes (4xx and 5xx) are treated as completed requests and passed to `on_complete`. To check the status, use `response.success?`, `response.client_error?`, or `response.server_error?`:
 
 ```ruby
-def on_complete(response)
-  if response.success?         # 2xx
-    process_data(response.json)
-  elsif response.client_error? # 4xx
-    handle_client_error(response)
-  elsif response.server_error? # 5xx
-    handle_server_error(response)
+class ApiCallback
+  def on_complete(response)
+    if response.success?
+      process_data(response.json)
+    elsif response.client_error?
+      handle_client_error(response.status, response.body)
+    elsif response.server_error?
+      handle_server_error(response.status, response.body)
+    end
+  end
+
+  def on_error(error)
+    Rails.logger.error("Request failed: #{error.message}")
   end
 end
 ```
 
-To treat non-2xx responses as errors instead, set `raise_error_responses: true` on the `RequestTask`:
+The `on_error` callback runs when the request raises an exception, such as a timeout or a connection failure. To treat HTTP errors as exceptions too, set the `raise_error_responses` option. With this option, a non-2xx response calls `on_error` with a `PatientHttp::HttpError`:
 
 ```ruby
-task = PatientHttp::RequestTask.new(
-  request: request,
-  task_handler: handler,
-  callback: "ApiCallback",
-  raise_error_responses: true
-)
+PatientHttp.get("https://api.example.com/data", callback: ApiCallback, raise_error_responses: true)
 ```
 
-When enabled, non-2xx responses call `TaskHandler#on_error` with an `HttpError` that provides access to the response:
+An `HttpError` gives you access to the request and the response:
 
 ```ruby
 def on_error(error)
   if error.is_a?(PatientHttp::HttpError)
-    puts error.status           # HTTP status code
-    puts error.url              # Request URL
-    puts error.response.body    # Response body
+    puts error.status              # HTTP status code
+    puts error.url                 # Request URL
+    puts error.http_method         # HTTP method
+    puts error.response.body       # Response body
+    puts error.response.headers    # Response headers
+    puts error.response.json       # Response body parsed as JSON
   end
 end
 ```
 
-## Request Templates
-
-For repeated requests to the same API, use `RequestTemplate` to share configuration:
-
-```ruby
-template = PatientHttp::RequestTemplate.new(
-  base_url: "https://api.example.com",
-  headers: {"Authorization" => "Bearer #{ENV['API_KEY']}"},
-  timeout: 60
-)
-
-# Build requests from the template
-get_request = template.get("/users/123")
-post_request = template.post("/users", json: {name: "John"})
-```
-
-Templates support all HTTP methods (`get`, `head`, `post`, `put`, `patch`, `delete`, `query`) and handle URL joining, header merging, and query parameter encoding.
+A 4xx response raises a `PatientHttp::ClientError`, and a 5xx response raises a `PatientHttp::ServerError`. Both are subclasses of `HttpError`.
 
-## Standard Interface
+To make `raise_error_responses` the default for every request, set it in the [configuration](#configuration).
 
-The `PatientHttp` module provides a standard interface for building and dispatching requests without needing to directly interact with the processor or task handlers. This allows you to write application code that makes HTTP requests without coupling it to the underlying async processing infrastructure.
+### Callback arguments
 
-You will need to register a request handler with `PatientHttp.register_handler` that defines how requests are dispatched to your job queue or background processing system. Once registered, you can use the `PatientHttp` class methods or the `RequestHelper` mixin to make async HTTP requests with callbacks.
+To pass data to your callbacks, use the `callback_args` option:
 
 ```ruby
-# The handler receives keyword arguments for the request, callback, and any additional callback arguments.
-PatientHttp.register_handler do |request:, callback:, callback_args: nil, raise_error_responses: nil|
-  # Example integration point. Adapt this to your app.
-  # Build a RequestTask and enqueue it to your processor.
-  task = PatientHttp::RequestTask.new(
-    request: request,
-    task_handler: MyTaskHandler.new,
-    callback: callback,
-    callback_args: callback_args,
-    raise_error_responses: raise_error_responses
-  )
-
-  processor.enqueue(task)
-end
-
-# Now you can make requests directly through the PatientHttp interface with the .request,
-# .get, .head, .post, .patch, .put, .delete, and .query class methods:
 PatientHttp.get(
-  "https://api.example.com/users/123",
+  "https://api.example.com/users/#{user_id}",
   callback: FetchUserCallback,
-  callback_args: {user_id: 123}
+  callback_args: {user_id: user_id, requested_at: Time.now.iso8601}
 )
 ```
 
-If you are using the [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) gem or the [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) gem, the appropriate handler will automatically be registered for you.
-
-### Inline Execution
-
-For consoles, tests, and development environments where no job system is configured, you can register a handler that executes requests inline — synchronously, in-process — instead of dispatching them to a queue:
+The `response.callback_args` and `error.callback_args` methods return the arguments:
 
 ```ruby
-PatientHttp.inline!
+response.callback_args[:user_id]
+response.callback_args["user_id"]
 ```
 
-Now every request made through the `PatientHttp` interface (or the `RequestHelper` mixin) runs immediately through the full request lifecycle (timeouts, redirects, error wrapping) and invokes its callback on the calling thread before returning. Callbacks can make further requests; those execute inline as well.
+The `callback_args` value follows these rules:
 
-```ruby
-PatientHttp.inline!
-PatientHttp.get("https://api.example.com/users/123", callback: FetchUserCallback)
-# FetchUserCallback#on_complete has already been invoked by this point
-```
+- It must be a Hash, or respond to `to_h`, and contain only JSON-native types: `nil`, `true`, `false`, `String`, `Integer`, `Float`, `Array`, and `Hash`.
+- Hash keys are converted to strings, including the keys of nested hashes and of hashes in arrays.
+- You can read the arguments with symbol or string keys: `callback_args[:user_id]` or `callback_args["user_id"]`.
+- Reading a key that isn't set raises a `KeyError`. To get a default value instead, use `callback_args.fetch(:user_id, nil)`.
 
-Inline requests run against `PatientHttp.default_configuration` by default (or a lazily created default configuration that includes any secrets registered with `PatientHttp.register_secret` — see [Secrets](#secrets)). You can also pass an explicit configuration:
+### Use request templates
+
+To share settings across requests to the same API, use `PatientHttp::RequestTemplate`:
 
 ```ruby
-PatientHttp.inline!(config: PatientHttp::Configuration.new(raise_error_responses: true))
-```
+template = PatientHttp::RequestTemplate.new(
+  base_url: "https://api.example.com",
+  headers: {"Authorization" => PatientHttp.secret(:api_token)},
+  timeout: 60
+)
 
-Use `PatientHttp.inline?` to check whether the inline handler is the currently registered handler. To execute a single request inline without registering a handler, use `PatientHttp.execute_inline(request:, callback:)`.
+request = template.get("/users/123")
+PatientHttp.execute(request: request, callback: FetchUserCallback)
+```
 
-### RequestHelper Mixin
+A template has a method for each HTTP method. The template joins each path with the base URL, and merges the headers and query parameters of each request with its own. If the template doesn't set a `timeout`, the configured `request_timeout` applies.
 
-Use `PatientHttp::RequestHelper` when you want a simple API for creating and dispatching async HTTP requests directly from your class.
+### Use the RequestHelper module
 
-1. Register a request handler with `PatientHttp.register_handler` that defines how requests are dispatched to your job queue or background processing system.
-2. Include `PatientHttp::RequestHelper` in your class.
-3. Optionally define a `request_template` for shared `base_url`, headers, and timeout.
-4. Call `async_get`, `async_head`, `async_post`, `async_put`, `async_patch`, `async_delete`, `async_query`, or `async_request`.
+For a class that makes many requests, include `PatientHttp::RequestHelper`. The module adds the `async_get`, `async_head`, `async_post`, `async_put`, `async_patch`, `async_delete`, `async_query`, and `async_request` methods. To set shared options such as `base_url`, `headers`, and `timeout`, use the `request_template` class method:
 
 ```ruby
 class ApiClient
@@ -237,435 +229,436 @@ class ApiClient
 
   request_template(
     base_url: "https://api.example.com",
-    headers: {"Authorization" => "Bearer #{ENV["API_KEY"]}"},
+    headers: {"Authorization" => PatientHttp.secret(:api_token)},
     timeout: 60
   )
 
   def fetch_user(user_id)
-    async_get(
-      "/users/#{user_id}",
-      callback: FetchUserCallback,
-      callback_args: {user_id: user_id}
-    )
+    async_get("/users/#{user_id}", callback: FetchUserCallback, callback_args: {user_id: user_id})
   end
 
   def update_user(user_id, data)
-    async_patch(
-      "/users/#{user_id}",
-      json: data,
-      callback: UpdateUserCallback,
-      callback_args: {user_id: user_id}
-    )
+    async_patch("/users/#{user_id}", json: data, callback: UpdateUserCallback, callback_args: {user_id: user_id})
   end
 end
 ```
 
-## Callback Arguments
+The `async_*` methods take the same options as the `PatientHttp` module methods. Paths are relative to the template's `base_url`. A subclass uses the template of its superclass unless it declares its own.
 
-Pass custom data through the request/response cycle using `callback_args`:
+## Configuration
 
-```ruby
-task = PatientHttp::RequestTask.new(
-  request: request,
-  task_handler: handler,
-  callback: "FetchDataCallback",
-  callback_args: {user_id: 123, request_timestamp: Time.now.iso8601}
-)
-```
+All configuration is optional. To set options, call `PatientHttp.configure` in an initializer. If an integration gem is loaded, the method yields that gem's configuration, which adds the options for the job system to the options below.
 
-Callback arguments are available on both `Response` and `Error` objects:
+Every call yields the same configuration object, so options accumulate. Several initializers can each set options without overwriting one another.
 
 ```ruby
-response.callback_args[:user_id]    # Symbol access
-response.callback_args["user_id"]   # String access
-```
-
-Callback args must contain only JSON-native types (`nil`, `true`, `false`, `String`, `Integer`, `Float`, `Array`, `Hash`). Hash keys are converted to strings for serialization.
+PatientHttp.configure do |config|
+  # Maximum concurrent HTTP requests (default: 256).
+  config.max_connections = 256
 
-## Response and Error Objects
+  # Maximum connections to each host (default: nil, no limit).
+  config.max_connections_per_host = 32
 
-The `PatientHttp::Response` and error objects are designed to be serializable and deserializable as JSON, making them safe to pass through job queues and across process boundaries. This allows you to enqueue the response or error data in your `TaskHandler` callbacks and process them asynchronously in another context.
+  # Default timeout for HTTP requests in seconds (default: 60).
+  config.request_timeout = 60
 
-Both response and error objects provide `as_json` and `to_json` methods for serialization:
+  # Timeout for graceful shutdown in seconds (default: 30).
+  config.shutdown_timeout = 30
 
-```ruby
-def on_complete(response, callback)
-  # Serialize the response for background processing
-  MyJobSystem.enqueue(callback, :on_complete, response.as_json)
-end
+  # Maximum response body size in bytes (default: 1MB). Larger responses raise
+  # ResponseTooLargeError.
+  config.max_response_size = 1024 * 1024
 
-def on_error(error, callback)
-  # Serialize the error for background processing
-  MyJobSystem.enqueue(callback, :on_error, error.as_json)
-end
-```
+  # Default User-Agent header for all requests (default: "PatientHttp").
+  config.user_agent = "MyApp/1.0"
 
-When deserializing, use the `load` class methods to reconstruct the objects:
+  # Whether to raise HttpError for non-2xx responses by default (default: false).
+  config.raise_error_responses = false
 
-```ruby
-response = PatientHttp::Response.load(json_data)
-error = PatientHttp::HttpError.load(json_data)
-```
+  # Maximum number of redirects to follow (default: 5; 0 turns off redirects).
+  config.max_redirects = 5
 
-The `Response` object includes the HTTP status code, headers, body, and callback arguments. Error objects (`HttpError`, `RedirectError`, `RequestError`) include the error message, context about the request, and callback arguments.
+  # Whether to follow redirects that change the HTTP method, such as POST to
+  # GET on a 302 (default: true). If false, the request gets the redirect
+  # response.
+  config.follow_method_changing_redirects = true
 
-Response headers are case insensitive. Headers that appear multiple times in the response (such as `set-cookie`) are flattened into a single joined string value.
+  # Names of headers to remove from all redirected requests (default: []).
+  # Names are case insensitive. Authorization and Cookie are always removed on
+  # cross-origin redirects.
+  config.redirect_strip_headers = ["X-Api-Key", "X-Internal-Token"]
 
-Response bodies are automatically encoded for JSON serialization. Binary content is Base64 encoded, and large text content is gzipped and then Base64 encoded to reduce payload size. Decoding is handled transparently when you access the `body` or `json` methods on the `Response` object.
+  # Maximum number of host clients to pool (default: 100).
+  config.connection_pool_size = 100
 
-### Payload Stores
+  # Timeout in seconds to open a connection, including the TCP connect and the
+  # TLS handshake (default: nil, no limit). It doesn't limit the wait for a
+  # response; request_timeout does that.
+  config.connection_timeout = 10
 
-For large request/response payloads, you can configure external storage to keep serialized JSON payloads small. Payloads exceeding the configured threshold are automatically stored externally and fetched on demand.
+  # TCP keepalive for pooled connections (default: nil, the kernel sends no
+  # probes). A number sets the idle seconds before the first probe. A Hash also
+  # sets the interval and the probe count, for example
+  # {idle: 30, interval: 10, count: 3}. The Hash must contain :idle. The
+  # :interval default is 10 seconds, and the :count default is 3 probes.
+  config.tcp_keepalive = 30
 
-If you are using a job queue or background processing system, this allows you to handle large requests or responses without hitting size limits or memory constraints on queue message payloads. The use of external storage is transparent to your application code.
+  # Seconds that sent data can stay unacknowledged before the kernel closes the
+  # connection (default: nil, the kernel default applies). Sets
+  # TCP_USER_TIMEOUT, which is available only on Linux.
+  config.tcp_user_timeout = 30
 
-```ruby
-# Register a payload store (see below for options; the file adapter should only be used for development/testing)
-config.register_payload_store(:my_store, adapter: :file, directory: "/tmp/payloads")
+  # HTTP or HTTPS proxy URL (default: nil). Supports authentication, for
+  # example "http://user:pass@proxy.example.com:8080".
+  config.proxy_url = "http://proxy.example.com:8080"
 
-# Use the ExternalStorage class to set and fetch stored payloads in your callbacks.
-storage = PatientHttp::ExternalStorage.new(config)
+  # Number of retries for failed requests (default: 3).
+  config.retries = 3
 
-large_response_data = storage.store(large_response.as_json)
-# Returns a reference like: {"$ref" => {"store" => "my_store", "key" => "abc123"}}
+  # HTTP protocol, :http1 or :http2 (default: nil, negotiated with the server,
+  # with HTTP/2 preferred for HTTPS). The :http1 value also limits the TLS ALPN
+  # advertisement to http/1.1, which can work around proxies that intercept SSL
+  # and don't handle HTTP/2 correctly.
+  config.protocol = nil
 
-small_response_data = storage.store(small_response.as_json, max_size: 1024)
-# Will not store the payload and returns the original data hash if the JSON payload is under 1KB.
+  # Number of threads that decode responses and deliver results (default: 2).
+  config.completion_threads = 2
 
-storage.storage_ref?(large_response_data) # => true
-storage.storage_ref?(small_response_data) # => false
+  # Number of times to retry result delivery before the failure is reported
+  # (default: 2).
+  config.completion_retries = 2
 
-storage.fetch(large_response_data) # Fetches the original data from the store
-storage.fetch(small_response_data) # Raises an error since this is not a reference
+  # Size in bytes above which payloads are stored externally when a payload
+  # store is configured (default: 64KB).
+  config.payload_store_threshold = 64 * 1024
 
-storage.delete(large_response_data) # Deletes the stored payload
+  # Logger (default: a Logger that writes errors to standard error).
+  config.logger = Rails.logger
+end
 ```
 
-#### File Store
+`PatientHttp.configuration` returns the same object outside a `configure` block.
 
-For development and testing:
+For the options that each job system adds, see the documentation for [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq#configuration) and [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue#configuration).
 
-```ruby
-config.register_payload_store(:files, adapter: :file, directory: "/tmp/payloads")
-```
+### Named processors
 
-#### Redis Store
+Named processors are available only when an integration gem is loaded. The integration gems add the `config.processor` method.
 
-For production with shared state across processes (requires the `redis` gem; the client must respond to `set`, `get`, `del`, and `exists`):
+By default, all requests share one processor and one `max_connections` limit. If one process runs workloads with very different profiles, such as slow LLM API calls and fast webhook deliveries, a burst of one workload can use all the capacity that the other needs. Named processor profiles keep the workloads separate:
 
 ```ruby
-redis = Redis.new(url: ENV["REDIS_URL"])
-config.register_payload_store(:redis, adapter: :redis, redis: redis, ttl: 86400)
+PatientHttp.configure do |config|
+  config.processor(:llm, max_connections: 200, request_timeout: 120)
+  config.processor(:webhooks, max_connections: 64, request_timeout: 10)
+end
 ```
 
-Options: `redis:` (required), `ttl:` (seconds, optional), `key_prefix:` (default: `"patient_http:payloads:"`)
-
-#### S3 Store
+Each profile runs as an independent processor in the process, with its own capacity, timeouts, and threads. Profile options override the top-level configuration. The profiles share every option that they don't override, such as secrets, preprocessors, payload stores, encryption, and the logger. The `:default` processor always exists.
 
-For durable storage across instances (requires `aws-sdk-s3` gem):
+To send a request to a processor, set the `processor:` option on the request method, on a `Request`, or on a `RequestTemplate`:
 
 ```ruby
-s3 = Aws::S3::Resource.new
-bucket = s3.bucket("my-payloads-bucket")
-config.register_payload_store(:s3, adapter: :s3, bucket: bucket)
+PatientHttp.post(url, callback: MyCallback, processor: :llm)
 ```
 
-Options: `bucket:` (required), `key_prefix:` (default: `"patient_http/payloads/"`)
+For how the processor name is kept through retries and crash recovery, see the documentation for your integration gem.
 
-#### ActiveRecord Store
+### Tuning tips
 
-For database-backed storage with transactional guarantees:
+- `max_connections`: Set this based on your system's resources. Each connection uses memory and a file descriptor. A tuned system with enough resources can handle thousands of concurrent connections.
+- `max_connections_per_host`: Limits the sockets open to each host. The default is no limit. For high concurrency, set a value such as 32, so that one host can't use every file descriptor. Make sure that the process file descriptor limit covers `max_connections`, plus idle pooled connections, plus the application's own connections.
+- `request_timeout`: Set this based on the response times of the APIs that you call. AI APIs can take minutes to respond while they generate content.
+- `connection_timeout`: Limits only the TCP connect and the TLS handshake. Set it to fail fast when a host doesn't answer. It doesn't limit the wait for a response, because `request_timeout` controls the full exchange.
+- `tcp_keepalive`: The kernel sends probes on an idle pooled connection. The probes keep NAT and firewall mappings open, and let the kernel find a dead peer before the pool sends a request on the connection. Set this option when connections stay idle in the pool between requests.
+- `tcp_user_timeout`: The kernel closes a connection when the peer doesn't acknowledge sent data. This option is available only on Linux. A request to a peer that stopped without notice fails after this timeout, without waiting for `request_timeout`. Acknowledged data isn't affected, so a slow response continues.
+- `connection_pool_size`: Sets the maximum number of hosts whose connections are kept open. Increase it if your application calls many different hosts.
+- `max_response_size`: Limits the size of HTTP responses to prevent high memory use from unexpectedly large responses. For a compressed response, the limit applies to the decompressed body. For large responses, consider a [payload store](#payload-stores).
+- `completion_threads`: Increase this when result delivery does heavy work, such as serialization or encryption, and finished requests wait for a thread. If the value is greater than 1, results are delivered concurrently, so `TaskHandler` callbacks and completion-time observers must be thread-safe. Set it to 1 to deliver results one at a time.
+- `completion_retries`: The number of times to retry result delivery before the failure is reported to observers through `completion_failed`. A retry calls `on_complete` or `on_error` again. As a result, a handler that raises an error *after* it enqueues its message delivers that message twice. Make handlers idempotent, or set `completion_retries` to 0 to report the first failure without a retry.
+- `shutdown_timeout`: Must be less than the process supervisor's stop timeout, so that in-flight requests and their results finish before a hard kill.
 
-```ruby
-config.register_payload_store(:database, adapter: :active_record)
-```
+### Response compression
 
-This requires a database migration. Copy the migration from the gem:
+Requests ask for `gzip` compression by default. A completion worker thread decompresses the response body. To change this, set the `accept-encoding` header on a request:
 
-```ruby
-# db/migrate/XXXXXX_create_patient_http_payloads.rb
-class CreatePatientHttpPayloads < ActiveRecord::Migration[7.0]
-  def change
-    create_table :patient_http_payloads, id: false do |t|
-      t.string :key, null: false, limit: 36
-      t.text :data, null: false
-      t.timestamps
-    end
+- `identity` turns off compression.
+- Any other encoding is delivered still encoded, with its `content-encoding` header, so that you can decode it yourself.
 
-    add_index :patient_http_payloads, :key, unique: true
-    add_index :patient_http_payloads, :created_at
-  end
-end
-```
+## Sensitive and large payloads
 
-Options: `model:` (optional, defaults to built-in `PatientHttp::PayloadStore::ActiveRecordStore::Payload`)
+Requests and responses are serialized into your job queue so that they can move between processes. As a result, you need to decide how to keep sensitive values out of the queue, and where to put payloads that are too large for it.
 
-#### Custom Stores
+These features cover both cases. They work together:
 
-Implement your own by subclassing `PatientHttp::PayloadStore::Base`:
+| Feature | When to use it | How to register it |
+| --- | --- | --- |
+| [Secrets](#secrets) | A sensitive header or query parameter, such as an API token. | `config.register_secret` |
+| [Preprocessors](#request-preprocessors) | A signature calculated from the final request, such as AWS SigV4. | `config.register_preprocessor` |
+| [Encryption](#encryption) | The request or response body is sensitive. | `config.encryption_key` |
+| [Payload stores](#payload-stores) | Payloads are too large for the queue. | `config.register_payload_store` |
 
-```ruby
-class MyStore < PatientHttp::PayloadStore::Base
-  register :my_store, self
+Use secrets and preprocessors first, because they keep sensitive values out of the queue. Use encryption when the payload itself is sensitive, and a payload store when the payload is too large.
 
-  def store(key, data)
-    # Store the hash and return the key
-  end
+### Secrets
 
-  def fetch(key)
-    # Return the hash or nil if not found
-  end
+If you put an API token directly on a request, the token is written to your job queue. A secret lets you refer to the value by name. The serialized request stores only a marker, `{"$secret" => "name"}`. The processor resolves the value from the configuration when it sends the request.
 
-  def delete(key)
-    # Delete the data (idempotent)
-  end
-end
+Register a secret with a value, or with a block that runs each time the secret is resolved:
 
-config.register_payload_store(:custom, adapter: :my_store, **options)
+```ruby
+PatientHttp.configure do |config|
+  config.register_secret(:authorization, "Bearer #{ENV["API_TOKEN"]}")
+  config.register_secret(:api_key) { ENV["MY_API_KEY"] }
+end
 ```
 
-Multiple stores can be registered for migration purposes. The last registered store is used for new writes; all registered stores remain available for reads.
-
-## Encryption
-
-When using PatientHttp with a job queue system, request and response data is serialized into the queue (Redis, database, etc.). If this data contains sensitive information, you should encrypt it.
+You can also register a secret at the module level. Use this in a library, or in an initializer that loads before the rest of your configuration:
 
-PatientHttp provides encryption helpers through the `Configuration` object, but it is up to the `TaskHandler` implementation to ensure that serialized data is actually encrypted. If you are using an integration gem like [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) or [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue), the `TaskHandler` provided by the gem handles encryption automatically — you just need to configure the encryption key or callables on the `Configuration` object.
-
-If you are writing a custom `TaskHandler`, use `Configuration#encryptor` as the helper and call `encrypt` / `decrypt` explicitly wherever your handler serializes or deserializes data.
+```ruby
+PatientHttp.register_secret(:api_key) { ENV["MY_API_KEY"] }
+```
 
-### Using an encryption key
+Module-level secrets are added to the configuration when it's created, so load order doesn't matter. To check whether a secret is registered, call `PatientHttp.secret_registered?(name)`.
 
-The simplest option is `encryption_key=`, which sets up [ActiveSupport::MessageEncryptor](https://api.rubyonrails.org/classes/ActiveSupport/MessageEncryptor.html) automatically using AES-256-GCM:
+Use a secret reference as a header or query parameter value:
 
 ```ruby
-config = PatientHttp::Configuration.new
-config.encryption_key = ENV["PATIENT_HTTP_ENCRYPTION_KEY"]
+PatientHttp.get(
+  "https://api.example.com/data",
+  callback: MyCallback,
+  headers: {"Authorization" => PatientHttp.secret(:authorization)},
+  params: {"api_key" => PatientHttp.secret(:api_key), "page" => 2}
+)
 ```
 
-To support key rotation, pass an array — the first key encrypts new data, and all keys attempt decryption:
+The secret query parameter isn't added to the serialized URL. The other parameters, such as `page`, are added as usual. The processor resolves both secrets immediately before it sends the request. If a secret isn't registered, the request fails with a `PatientHttp::SecretManager::SecretNotFoundError`, which is passed to `on_error`.
 
-```ruby
-config.encryption_key = [ENV["PATIENT_HTTP_ENCRYPTION_KEY"], ENV["PATIENT_HTTP_OLD_KEY"]]
-```
+### Request preprocessors
 
-### Using custom callables
+A preprocessor changes a request immediately before it's sent, usually to sign it. Signing schemes such as AWS SigV4 calculate values from the final request and set several headers, so a static header value can't express them.
 
-For custom encryption libraries, provide callables that accept and return raw bytes (String):
+Like secrets, preprocessors are registered in the configuration and referenced by name. The signing code and its credentials stay in the processor and aren't written to the queue.
 
 ```ruby
-config.encryption { |bytes| MyEncryption.encrypt(bytes) }
-config.decryption { |bytes| MyEncryption.decrypt(bytes) }
+PatientHttp.configure do |config|
+  config.register_preprocessor(:aws_sigv4) do |request|
+    signer = Aws::Sigv4::Signer.new(
+      service: "execute-api",
+      region: "us-east-1",
+      credentials_provider: Aws::CredentialProviderChain.new.resolve
+    )
+    signature = signer.sign_request(
+      http_method: request.http_method.to_s.upcase,
+      url: request.url,
+      headers: request.headers.to_h,
+      body: request.body.to_s
+    )
+    signature.headers.each { |name, value| request.headers[name] = value }
+  end
+end
+
+PatientHttp.post("https://api.example.com/data", callback: MyCallback, json: {value: 1}, preprocessors: :aws_sigv4)
 ```
 
-Or pass any object that responds to `#call`:
+The preprocessor receives a `PatientHttp::OutgoingRequest`. At that time, secret references are resolved, and the `x-request-id` and default `User-Agent` headers are set. The object has these methods:
 
-```ruby
-config.encryption(->(bytes) { MyEncryption.encrypt(bytes) })
-config.decryption(->(bytes) { MyEncryption.decrypt(bytes) })
-```
+- `http_method`, `url`, and `body`: Read-only. The URL includes the resolved secret query parameters.
+- `headers`: The request headers. You can change them. Names are case insensitive.
+- `add_param(name, value)`: Adds a query parameter to the URL, for signing schemes that use query parameters.
 
-### Wiring encryption into a custom TaskHandler
+To run several preprocessors, pass an array. They run in order, and each one sees the changes of the ones before it. `RequestTemplate` and `request_template` in `RequestHelper` take a `preprocessors:` option as a default. If a preprocessor isn't registered, the request fails with a `PatientHttp::RequestPreparer::PreprocessorNotFoundError`, which is passed to `on_error`.
 
-If you are writing your own `TaskHandler` (rather than using one from an integration gem), you must wire in encryption yourself. `Configuration#encryptor` returns an `Encryptor` built from the configured callables. Call it directly at every serialization boundary:
+When a request follows a same-origin redirect, the preprocessors run again for the redirect URL, so signatures stay valid. On a cross-origin redirect, the preprocessors are removed, as are the `Authorization` and `Cookie` headers. As a result, signed credentials aren't sent to another origin.
 
-```ruby
-class MyTaskHandler < PatientHttp::TaskHandler
-  def initialize(job_id, configuration:)
-    @job_id = job_id
-    @configuration = configuration
-  end
+### Encryption
 
-  def on_complete(response, callback)
-    # Encrypt the serialized response before enqueuing
-    encrypted = @configuration.encryptor.encrypt(response.as_json)
-    MyJobSystem.enqueue(callback, :on_complete, encrypted)
-  end
+When the request or response body is sensitive, encrypt it. After you set a key, the integration gems encrypt data before they write it to the queue, and decrypt it when they read it.
 
-  def on_error(error, callback)
-    encrypted = @configuration.encryptor.encrypt(error.as_json)
-    MyJobSystem.enqueue(callback, :on_error, encrypted)
-  end
+The simplest option is `encryption_key`. It uses [ActiveSupport::MessageEncryptor](https://api.rubyonrails.org/classes/ActiveSupport/MessageEncryptor.html) with AES-256-GCM:
 
-  def retry
-    MyJobSystem.enqueue_job(@job_id)
-  end
+```ruby
+PatientHttp.configure do |config|
+  config.encryption_key = ENV["PATIENT_HTTP_ENCRYPTION_KEY"]
 end
-
-# Keep a configuration reference and use config.encryptor where needed
-handler = MyTaskHandler.new("job-123", configuration: config)
 ```
 
-In your callback, decrypt before processing:
+To rotate keys, pass an array. The first key encrypts data, and all keys are tried for decryption:
 
 ```ruby
-class FetchDataCallback
-  def initialize(configuration:)
-    @configuration = configuration
-  end
-
-  def on_complete(data)
-    response = PatientHttp::Response.load(@configuration.encryptor.decrypt(data))
-    # ...
-  end
+PatientHttp.configure do |config|
+  config.encryption_key = [ENV["PATIENT_HTTP_ENCRYPTION_KEY"], ENV["PATIENT_HTTP_OLD_KEY"]]
 end
 ```
 
-### How it works
+To use another encryption library, provide callables that take and return raw bytes as a String:
 
-Encrypted data is stored as `{"__encrypted__" => true, "value" => "<base64>"}`. The `Encryptor` JSON-serializes the original hash, passes the bytes to your callable, and Base64-encodes the result. Decryption reverses the process. Hashes without the `"__encrypted__"` key are passed through unchanged, so un-encrypted historical data continues to work while you roll out encryption.
+```ruby
+PatientHttp.configure do |config|
+  config.encryption { |bytes| MyEncryption.encrypt(bytes) }
+  config.decryption { |bytes| MyEncryption.decrypt(bytes) }
+end
+```
 
-## Secrets
+You can also pass any object that responds to `call`.
 
-Requests are serialized into your job queue before they run. If you put a sensitive value — an API token in an `Authorization` header, or an API key in a query parameter — directly on the request, that value is written into the queue. Requests can be encrypted in the queue, but a better practice is to avoid putting sensitive values on the request at all.
+Encrypted data is stored as `{"__encrypted__" => true, "value" => "<base64>"}`. The `Encryptor` serializes the original hash to JSON, passes the bytes to your callable, and encodes the result with Base64. A hash without the `"__encrypted__"` key is returned unchanged, so data that was written before you turned on encryption can still be read.
 
-The secret manager lets you reference a sensitive values in headers or query parameters by name instead. The serialized request stores only a reference marker (`{"$secret" => "name"}`), never the value. The actual value lives on the `Configuration` (which exists on the processor side) and is resolved at the moment the request is sent.
+If you write your own `TaskHandler`, see [Build a custom integration](#build-a-custom-integration) for how to add encryption.
 
-### Defining secrets
+### Payload stores
 
-Register named secrets on the `Configuration`. A value can be given directly, or as a block that is evaluated lazily each time the secret is resolved (useful for reading from the environment on demand):
+When you register a payload store, any serialized payload larger than `payload_store_threshold` is written to the store instead of the job queue. The queue gets a small reference, which is resolved when the payload is needed. Queue messages stay small, and your application code doesn't change.
 
 ```ruby
-config = PatientHttp::Configuration.new
-config.register_secret(:authorization, "Bearer #{ENV['API_TOKEN']}") # static value
-config.register_secret(:api_key) { ENV["MY_API_KEY"] } # lazy block
+PatientHttp.configure do |config|
+  config.register_payload_store(:redis, adapter: :redis, redis: Redis.new(url: ENV["REDIS_URL"]), ttl: 86_400)
+  config.payload_store_threshold = 64 * 1024
+end
 ```
 
-If a secret is not found when resolving a request, a `PatientHttp::SecretManager::SecretNotFoundError` is raised, which surfaces through the normal request error path.
+These adapters are available:
 
-#### Module-level registration
+| Adapter | Options | Notes |
+| --- | --- | --- |
+| `:file` | `directory:` | For development and tests only. Hosts don't share the files. |
+| `:redis` | `redis:` (required), `ttl:`, `key_prefix:` (default `"patient_http:payloads:"`) | Requires the `redis` gem. The client must respond to `set`, `get`, `del`, and `exists`. |
+| `:s3` | `bucket:` (required), `key_prefix:` (default `"patient_http/payloads/"`) | Requires the `aws-sdk-s3` gem. |
+| `:active_record` | `model:` | Requires a database table. See the following section. |
 
-If the `Configuration` is owned by an integration gem (patient_http-sidekiq, patient_http-solid_queue), your application code may not have a convenient reference to it — or may load before it exists. In that case, register secrets at the module level instead:
+The Active Record adapter needs the `patient_http_payloads` table. In a Rails app, load the engine in an initializer, and then install and run the migration:
 
 ```ruby
-PatientHttp.register_secret(:authorization, "Bearer #{ENV['API_TOKEN']}")
-PatientHttp.register_secret(:api_key) { ENV["MY_API_KEY"] }
+require "patient_http/rails/engine"
 ```
 
-Module-level secrets are applied to `PatientHttp.default_configuration` — immediately if one is already set, or as soon as one is set later — so registration order between your application code and the integration gem's configuration does not matter. Integration gems set the default configuration at the end of their configure step; you can also set it yourself:
-
-```ruby
-PatientHttp.default_configuration = config
+```bash
+bin/rails patient_http:install:migrations
+bin/rails db:migrate
 ```
 
-Use `PatientHttp.secret_registered?(name)` to check whether a secret is available, either at the module level or on the default configuration.
-
-### Referencing secrets when building a request
-
-Use `PatientHttp.secret(name)` anywhere you would put a sensitive header or query parameter value. No value is needed (or available) at build time:
+You can also add the migration yourself:
 
 ```ruby
-PatientHttp.get(
-  "https://api.example.com/data",
-  callback: MyCallback,
-  headers: {"Authorization" => PatientHttp.secret(:api_token)},
-  params: {"api_key" => PatientHttp.secret(:api_key), "page" => 2}
-)
-```
+class CreatePatientHttpPayloads < ActiveRecord::Migration[7.0]
+  def change
+    create_table :patient_http_payloads, id: false do |t|
+      t.string :key, null: false, limit: 36
+      t.text :data, null: false
+      t.timestamps
+    end
 
-The request serializes the secret header as `{"$secret" => "api_token"}` and keeps the secret query parameter out of the URL (non-secret params like `page` are still folded into the URL as usual). The processor dereferences both just before sending: the header is set to its resolved value and the resolved query parameter is appended to the URL.
+    add_index :patient_http_payloads, :key, unique: true
+    add_index :patient_http_payloads, :created_at
+  end
+end
+```
 
-## Request Preprocessors
+To write your own adapter, subclass `PatientHttp::PayloadStore::Base`, and implement `store_json`, `fetch`, and `delete`. The adapter must be thread-safe.
 
-Preprocessors let you modify a request just before it is sent — most usefully, to sign it. Signing schemes like AWS SigV4 need to compute values over the final outgoing request (method, URL, headers, body) and set multiple headers, which cannot be expressed as a static header value at build time.
+```ruby
+class MyStore < PatientHttp::PayloadStore::Base
+  register :my_store, self
 
-Like secrets, preprocessors are registered on the `Configuration` and referenced from requests by name only. The serialized request carries just the name, so the signing logic and its credentials live on the processor side and are never written to the job queue.
+  def store_json(key, json)
+    # Store the JSON string and return the key.
+  end
 
-### Defining preprocessors
+  def fetch(key)
+    # Return the parsed hash, or nil if the key isn't found.
+  end
 
-Register a named preprocessor as a block or callable taking a single argument:
+  def delete(key)
+    # Delete the data. Don't raise an error if the key doesn't exist.
+  end
+end
 
-```ruby
-config = PatientHttp::Configuration.new
-config.register_preprocessor(:aws_sigv4) do |request|
-  signer = Aws::Sigv4::Signer.new(
-    service: "execute-api",
-    region: "us-east-1",
-    credentials_provider: Aws::CredentialProviderChain.new.resolve
-  )
-  signature = signer.sign_request(
-    http_method: request.http_method.to_s.upcase,
-    url: request.url,
-    headers: request.headers.to_h,
-    body: request.body.to_s
-  )
-  signature.headers.each { |name, value| request.headers[name] = value }
+PatientHttp.configure do |config|
+  config.register_payload_store(:custom, adapter: :my_store, **options)
 end
 ```
 
-The argument is a `PatientHttp::OutgoingRequest` — a view of the request as it is about to be sent, after all secret references have been resolved and the `x-request-id` and default `User-Agent` headers have been set. It exposes:
-
-- `http_method`, `url`, and `body` (read-only; the URL includes any resolved secret query params)
-- `headers` — mutable, case-insensitive headers
-- `add_param(name, value)` — appends a query parameter to the URL, for signed-query-param schemes
-
-### Attaching preprocessors to a request
+To move to a new store, register both stores. The last store registered is used for new writes. All registered stores are available for reads.
 
-Reference registered preprocessors by name when building a request:
+To store and fetch payloads yourself, use `PatientHttp::ExternalStorage`:
 
 ```ruby
-PatientHttp.post(
-  "https://api.example.com/data",
-  callback: MyCallback,
-  json: {value: 1},
-  preprocessors: :aws_sigv4
-)
-```
-
-Multiple preprocessors can be given as an array; they run in order, each seeing the changes made by the ones before it. `RequestTemplate` and the `RequestHelper` mixin's `request_template` also accept `preprocessors:` as a template-wide default, and the mixin's `async_*` helpers accept `preprocessors:` per request.
-
-If a request references a preprocessor name that is not registered, a `PatientHttp::RequestPreparer::PreprocessorNotFoundError` is raised, which surfaces through the normal request error path.
+storage = PatientHttp::ExternalStorage.new(PatientHttp.configuration)
 
-When redirects are followed, preprocessors are re-run against each redirect URL so signatures stay valid. On cross-origin redirects they are dropped entirely, consistent with the stripping of `Authorization` and `Cookie` headers, so signed credentials are never sent to an unexpected origin.
+data = storage.store(response.as_json, max_size: 1024) # Returns the original hash if it's 1KB or smaller.
+storage.storage_ref?(data)                             # Returns true if the hash was stored.
+storage.fetch(data)                                    # Returns the original hash.
+storage.delete(data)                                   # Deletes the stored payload.
+```
 
 ## Redirects
 
-Redirect responses (300, 301, 302, 303, 307, and 308) with a `Location` header are followed automatically, up to `max_redirects` hops. A 300 response is followed only when the server names a preferred choice in `Location`. Redirect loops raise `RecursiveRedirectError` and exceeding the limit raises `TooManyRedirectsError`. Any redirect that is not followed is delivered to the callback as a normal response.
+A redirect response (300, 301, 302, 303, 307, or 308) with a `Location` header is followed automatically, up to `max_redirects` times. A 300 response is followed only when the `Location` header names the server's preferred choice. A redirect loop raises a `RecursiveRedirectError`, and too many redirects raise a `TooManyRedirectsError`. A redirect that isn't followed is passed to the callback as a normal response.
 
 The HTTP method of the redirected request follows RFC 9110:
 
 | Status | Method |
 | --- | --- |
-| 301, 302 | `POST` becomes `GET` and the body is dropped. Other methods (including `HEAD`, `PUT`, `DELETE`, and `QUERY`) are preserved with their body. |
-| 303 | `GET` and `HEAD` are preserved. Every other method becomes `GET` and the body is dropped. |
-| 300, 307, 308 | The method and body are preserved. |
+| 301, 302 | `POST` changes to `GET`, and the body is removed. Other methods, including `HEAD`, `PUT`, `DELETE`, and `QUERY`, keep their method and body. |
+| 303 | `GET` and `HEAD` keep their method. All other methods change to `GET`, and the body is removed. |
+| 300, 307, 308 | The method and body don't change. |
+
+The QUERY specification states that the POST-to-GET exception for 301 and 302 doesn't apply to `QUERY`. As a result, a redirected `QUERY` is sent again as a `QUERY` with its body, and a 303 changes it to a `GET`.
 
-The QUERY specification states that the POST-to-GET exception on 301 and 302 does not apply to `QUERY`, so a redirected `QUERY` is re-sent as a `QUERY` with its body, and a 303 turns it into a `GET`.
+### Prevent method changes
 
-### Preventing method changes
+To stop following redirects that change the HTTP method, set `follow_method_changing_redirects: false`. For example, a `POST` that gets a 302 then completes with the 302 response, instead of a `GET` to the new location. Redirects that keep the method are still followed, such as a `PUT` that gets a 301, or any method that gets a 307.
 
-Set `follow_method_changing_redirects: false` to stop following redirects that would change the HTTP method. A `POST` that receives a 302 then completes with the 302 response instead of being retried as a `GET`. Redirects that preserve the method (a `PUT` on a 301, or any method on a 307) are still followed. The option can be set on the `Configuration` or on a single `Request`; the request value wins when both are set.
+You can set the option in the configuration or on a request. If both are set, the request value applies.
 
 ```ruby
-config = PatientHttp::Configuration.new(follow_method_changing_redirects: false)
+PatientHttp.configure do |config|
+  config.follow_method_changing_redirects = false
+end
 
-# Or per request
-request = PatientHttp::Request.new(:post, "https://api.example.com/submit", body: payload, follow_method_changing_redirects: false)
+# Or for one request.
+PatientHttp.post(url, callback: MyCallback, body: payload, follow_method_changing_redirects: false)
 ```
 
-### Stripping headers on redirects
+### Remove headers on redirects
 
-`Authorization` and `Cookie` headers are always removed on cross-origin redirects. To make sure other sensitive headers are never sent to a redirect target, list them in `redirect_strip_headers`. Header names are matched case insensitively. Listed headers are removed from every redirected request, same-origin or not.
+The `Authorization` and `Cookie` headers are always removed on cross-origin redirects. To keep other sensitive headers from being sent to a redirect target, list them in `redirect_strip_headers`. Header names are case insensitive. The listed headers are removed from every redirected request, same-origin or cross-origin.
 
 ```ruby
-config = PatientHttp::Configuration.new(redirect_strip_headers: ["X-Api-Key", "X-Internal-Token"])
-
-# Or per request; these are stripped in addition to the configured headers
-request = PatientHttp::Request.new(:get, "https://api.example.com/data", headers: headers, redirect_strip_headers: "X-Signature")
+PatientHttp.configure do |config|
+  config.redirect_strip_headers = ["X-Api-Key", "X-Internal-Token"]
+end
 
-# The same options are accepted by PatientHttp.request, the async_* helpers, and RequestTemplate
+# Or for one request. These headers are removed in addition to the configured headers.
 PatientHttp.get("https://api.example.com/data", callback: FetchCallback, redirect_strip_headers: "X-Signature")
 ```
 
-Per-request header names survive serialization into the job queue, so they apply no matter which process follows the redirect.
+The header names for a request are serialized with it into the job queue, so they apply in the process that follows the redirect.
 
-Stripping applies to the headers set on the request. Preprocessors run again on each same-origin redirect and can add headers after the strip, so a header that a preprocessor sets is sent to the redirect target. When a redirect changes the method and drops the body, the headers that describe the body (`Content-Type`, `Content-Length`, `Content-Encoding`, `Content-Language`, and `Content-Location`) are removed as well.
+Only the headers set on the request are removed. Preprocessors run again on each same-origin redirect and can add headers after they're removed. As a result, a header that a preprocessor sets is sent to the redirect target.
+
+When a redirect changes the method and removes the body, the headers that describe the body are removed as well: `Content-Type`, `Content-Length`, `Content-Encoding`, `Content-Language`, and `Content-Location`.
+
+## Response and error objects
+
+`PatientHttp::Response` and the error objects can be serialized to JSON, so they can move through job queues and between processes. Both have `as_json` and `to_json` methods. To create an object from JSON data, use the `load` class method:
+
+```ruby
+response = PatientHttp::Response.load(json_data)
+error = PatientHttp::Error.load(json_data)
+```
+
+A `Response` has the HTTP status code, headers, body, and callback arguments. The error classes, `HttpError`, `RedirectError`, and `RequestError`, have the error message, details about the request, and callback arguments. All error classes are subclasses of `PatientHttp::Error`, and `error.error_type` returns a symbol that identifies the kind of error, such as `:timeout`, `:connection`, or `:http_error`.
+
+Request and response header names are case insensitive. A request header with a `nil` or empty value is never sent. If you set a header to `nil` or `""`, the header is removed, and a header Hash such as `{"X-Header" => nil}` doesn't set the header. A response header that occurs more than one time, such as `set-cookie`, becomes one string with the values joined.
+
+Response bodies are encoded for JSON serialization. Binary content is encoded with Base64. Large text content is compressed with gzip and then encoded with Base64 to reduce the payload size. The `body` and `json` methods of `Response` decode the body for you.
 
 ## Troubleshooting
 
 ### Warning: `ThreadError: Attempt to unlock a mutex which is not locked`
 
-On some Ruby versions you may see a warning like this in your logs:
+On some Ruby versions, a warning like this can appear in your logs:
 
 ```
 warn: Async::Task: Async::Pool::Controller Gardener [...]
@@ -674,114 +667,122 @@ warn: Async::Task: Async::Pool::Controller Gardener [...]
     |   → .../async-pool-x.y.z/lib/async/pool/controller.rb:132 in `synchronize'
 ```
 
-This is caused by [Ruby bug #20907](https://bugs.ruby-lang.org/issues/20907) (see also [socketry/async#424](https://github.com/socketry/async/issues/424)): under the fiber scheduler, a fiber interrupted while waiting on a `ConditionVariable` fails to re-acquire its mutex before unwinding, raising a spurious `ThreadError`. It appears whenever a pooled HTTP client is closed while its connection pool's background "gardener" task is idle — for example when a connection is evicted after a connection error, when the least recently used client is evicted because the pool is full, or when the processor shuts down.
+The cause is [Ruby bug #20907](https://bugs.ruby-lang.org/issues/20907). For more information, see [socketry/async#424](https://github.com/socketry/async/issues/424). With the fiber scheduler, a fiber that's interrupted while it waits on a `ConditionVariable` doesn't get its mutex again before it exits, which raises a false `ThreadError`. The warning appears when a pooled HTTP client closes while its connection pool's background gardener task is idle. For example, this happens when the pool evicts a connection after a connection error, when the pool is full and evicts the least recently used client, or when the processor shuts down.
 
-The warning is harmless — connections are still closed correctly; only the log noise is wrong. The fix is to upgrade Ruby: the bug is fixed in Ruby 3.2.7+, 3.3.7+, and 3.4+.
+The warning is harmless. The connections still close correctly. To remove the warning, upgrade Ruby. The bug is fixed in Ruby 3.2.7 and later, 3.3.7 and later, and 3.4 and later.
 
-## Configuration
+## Build a custom integration
 
-```ruby
-config = PatientHttp::Configuration.new(
-  # Maximum concurrent HTTP requests (default: 256)
-  max_connections: 256,
+This section explains how to integrate the gem with a job system that doesn't have an integration gem. If you use Sidekiq or Solid Queue, the integration gems do all of this for you.
 
-  # Default timeout for HTTP requests in seconds (default: 60)
-  request_timeout: 60,
+An integration has these parts:
 
-  # Timeout for graceful shutdown in seconds (default: 30)
-  shutdown_timeout: 30,
+- A `TaskHandler` that delivers results to your job system.
+- A `Processor` that runs requests.
+- A registered request handler, so that application code can use the `PatientHttp` module methods.
 
-  # Maximum response body size in bytes (default: 1MB)
-  max_response_size: 1024 * 1024,
+### Create a task handler
 
-  # Default User-Agent header (default: "PatientHttp")
-  user_agent: "MyApp/1.0",
+The `TaskHandler` connects the processor to your job system:
 
-  # Treat non-2xx responses as errors by default (default: false)
-  raise_error_responses: false,
+```ruby
+class MyTaskHandler < PatientHttp::TaskHandler
+  def initialize(job_id)
+    @job_id = job_id
+  end
 
-  # Maximum redirects to follow (default: 5, 0 disables)
-  max_redirects: 5,
+  def on_complete(response, callback)
+    MyJobSystem.enqueue(callback, :on_complete, response.as_json)
+  end
 
-  # Follow redirects that must change the HTTP method, such as POST to GET on
-  # a 302 (default: true). When false, those requests receive the redirect response.
-  follow_method_changing_redirects: true,
+  def on_error(error, callback)
+    MyJobSystem.enqueue(callback, :on_error, error.as_json)
+  end
+
+  def retry
+    # Re-enqueue the original job when the processor shuts down before the
+    # request finishes.
+    MyJobSystem.enqueue_job(@job_id)
+  end
+end
+```
 
-  # Header names (case insensitive) always stripped from redirected requests
-  # (default: []). Authorization and Cookie are always stripped on cross-origin
-  # redirects.
-  redirect_strip_headers: ["X-Api-Key", "X-Internal-Token"],
+> [!IMPORTANT]
+> Task handler callbacks run on the processor's completion worker threads (see `completion_threads`), not on the reactor thread, so they don't block the event loop. Keep them lightweight anyway. Usually, a callback only enqueues a message for another system. Heavy callbacks compete with the reactor for the GVL. A task also counts against capacity until its result is delivered, so slow callbacks reduce request capacity.
+>
+> Callbacks must be thread-safe. By default, two completion worker threads deliver results concurrently. As a result, two callbacks can run at the same time, in an order unrelated to the order in which the requests finished. To deliver results one at a time, set `completion_threads` to 1.
+>
+> Callbacks must also be idempotent. A callback that raises an error is retried `completion_retries` times (default 2). If a callback raises an error after it enqueues its message, the message is enqueued again. If that isn't acceptable, set `completion_retries` to 0.
 
-  # Maximum number of hosts to maintain persistent connections for (default: 100)
-  connection_pool_size: 100,
+The task handler is responsible for encryption. The base class doesn't encrypt or decrypt data. Use `Configuration#encryptor` each time you serialize data:
 
-  # Connection timeout in seconds (default: nil, uses request_timeout)
-  connection_timeout: 10,
+```ruby
+def on_complete(response, callback)
+  encrypted = @configuration.encryptor.encrypt(response.as_json)
+  MyJobSystem.enqueue(callback, :on_complete, encrypted)
+end
+```
 
-  # HTTP/HTTPS proxy URL (default: nil)
-  proxy_url: "http://proxy.example.com:8080",
+Decrypt the data before you create the object again:
 
-  # Retries for failed requests (default: 3)
-  retries: 3,
+```ruby
+response = PatientHttp::Response.load(@configuration.encryptor.decrypt(data))
+```
 
-  # Force the HTTP protocol to :http1 or :http2 (default: nil, negotiates with
-  # the server, preferring HTTP/2 for HTTPS). Forcing :http1 also limits the TLS
-  # ALPN advertisement to http/1.1, which can work around SSL-intercepting
-  # proxies that mishandle HTTP/2.
-  protocol: nil,
+### Run a processor
 
-  # Logger instance (default: Logger to STDERR at ERROR level)
-  logger: Logger.new($stdout)
-)
+```ruby
+config = PatientHttp::Configuration.new(max_connections: 256, request_timeout: 60)
+processor = PatientHttp::Processor.new(config)
+processor.start
 
-# Register named secrets to reference sensitive headers/params indirectly (see Secrets)
-config.register_secret(:api_token, ENV["MY_API_TOKEN"])
+task = PatientHttp::RequestTask.new(
+  request: PatientHttp::Request.new(:get, "https://api.example.com/users/123"),
+  task_handler: MyTaskHandler.new("job-123"),
+  callback: "FetchUserCallback",
+  callback_args: {user_id: 123}
+)
+processor.enqueue(task)
 ```
 
-### Tuning Tips
+If the processor isn't running, `enqueue` raises a `PatientHttp::NotRunningError`. If the processor is at `max_connections`, it raises a `PatientHttp::MaxCapacityError`.
 
-- **max_connections**: Each connection uses memory and file descriptors. A tuned system can handle thousands.
-- **max_connections_per_host**: Bounds sockets per host (default unlimited). Set a value such as 32 for high-concurrency deployments so one host cannot consume every file descriptor. Verify the process file descriptor limit covers `max_connections` plus pooled idle host connections plus the application's own connections.
-- **request_timeout**: Set based on expected API response times. AI/LLM APIs may need minutes.
-- **connection_pool_size**: Increase for applications calling many different API hosts.
-- **max_response_size**: Keeps memory usage bounded. Large responses may need external payload storage. The limit applies to the inflated bytes of compressed responses.
-- **Response compression**: Requests ask for `gzip` by default and the body is inflated on a completion worker thread. Set `accept-encoding` on a request to change this: `identity` skips compression, and any other encoding is delivered still encoded with its `content-encoding` header kept so you can decode it yourself.
-- **completion_threads**: Number of threads that decode responses and deliver results (default 2). Increase when callbacks do heavier work (serialization, encryption) and completions back up behind them. Any value above 1 delivers results concurrently, so `TaskHandler` callbacks and completion-time observers must be thread-safe. Use 1 to serialize delivery.
-- **completion_retries**: Delivery retries before a result is reported through `completion_failed` (default 2). A retry calls `on_complete`/`on_error` again, so a handler that raises *after* enqueuing its message delivers that message twice. Make handlers idempotent, or set `completion_retries: 0` to report the first failure without retrying.
-- **shutdown_timeout**: Set below the process supervisor's termination window so the drain (including handed-off completions) finishes before a hard kill.
+A process can run several named processors, each with its own capacity and threads:
 
-## Processor Lifecycle
+```ruby
+PatientHttp::Processor.new(config, name: :llm)
+```
+
+A request has an optional `processor` name, which is serialized with the request. Integrations use this name to send the request to a processor.
 
-The processor transitions through these states:
+### Processor lifecycle
 
 ```
 stopped -> starting -> running -> draining -> stopping -> stopped
 ```
 
-- **stopped**: Not processing requests
-- **starting**: Initializing the reactor thread
-- **running**: Accepting and processing requests
-- **draining**: Rejecting new requests, completing in-flight ones
-- **stopping**: Shutting down, re-enqueuing incomplete requests
+- `stopped`: The processor doesn't run requests.
+- `starting`: The processor starts the reactor thread.
+- `running`: The processor accepts and runs requests.
+- `draining`: The processor rejects new requests and finishes in-flight requests.
+- `stopping`: The processor shuts down and re-enqueues requests that didn't finish.
 
 ```ruby
-processor = PatientHttp::Processor.new(config)
-
-processor.start              # Start processing
+processor.start              # Start the processor.
 processor.running?           # => true
 
-processor.drain              # Stop accepting new requests
+processor.drain              # Stop accepting new requests.
 processor.draining?          # => true
 
-processor.stop(timeout: 25)  # Graceful shutdown
+processor.stop(timeout: 25)  # Shut down gracefully.
 processor.stopped?           # => true
 ```
 
-When the processor stops with in-flight requests, it calls `TaskHandler#retry` on each incomplete task so they can be re-enqueued.
+When the processor stops with in-flight requests, it calls `TaskHandler#retry` for each request that didn't finish, so that the job system can enqueue it again.
 
-### Observing the Processor
+### Observe the processor
 
-Register observers to monitor processor events:
+To receive processor events, subclass `PatientHttp::ProcessorObserver`, and override the methods for the events that you need:
 
 ```ruby
 class MetricsObserver < PatientHttp::ProcessorObserver
@@ -805,27 +806,48 @@ end
 processor.observe(MetricsObserver.new)
 ```
 
-Observers can also track the full task pipeline:
+Observers can also track each task through the processor:
 
- - `request_enqueued(request_task)` is called when a task is announced to the processor (before the task is visible to the reactor). It is guaranteed to arrive before `request_start`, so observers can set up durable tracking (e.g. a crash-recovery registry entry) before `Processor#enqueue` returns or raises.
- - `request_rejected(request_task)` is called when an announced task is not accepted (not running or at capacity), so observers can tear down anything they set up in `request_enqueued`.
-- `request_requeued(request_task)` is called when an incomplete task is re-enqueued through its task handler (processor shutdown or reactor failure). The job system owns the request again once this is sent.
+- `request_enqueued(request_task)`: Runs when a task is given to the processor, before the reactor can see it. This method always runs before `request_start`. As a result, observers can set up durable tracking, such as a crash-recovery registry entry, before `Processor#enqueue` returns or raises an error.
+- `request_rejected(request_task)`: Runs when the processor doesn't accept a task, because it isn't running or is at capacity. Remove anything that `request_enqueued` set up.
+- `request_requeued(request_task)`: Runs when a task that didn't finish is re-enqueued through its task handler, because the processor shut down or the reactor failed. After this call, the job system owns the request again.
+- `completion_failed(request_task, error)`: Runs when a result can't be delivered after all retries. `request_end` doesn't run in this case, so durable tracking stays in place for external recovery.
 
-Use `Processor#tracked_request_ids` to get the IDs of all tasks in the pipeline (queued, pending, and in-flight), for example to keep heartbeats alive for tasks that have not started yet.
+To get the IDs of all queued, pending, and in-flight tasks, call `Processor#tracked_request_ids`. For example, use it to update heartbeats for tasks that haven't started yet.
 
-## Testing
+For the thread that runs each observer method, see the `ProcessorObserver` documentation. Observers must be thread-safe.
 
-Use `SynchronousExecutor` to execute requests synchronously in tests. This class can be used in place of the async processor for testing your request handling logic without needing to start the full async infrastructure.
+### Register a request handler
 
-It is integrated automatically in the [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) and [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) gems.
+To let application code use the `PatientHttp` module methods, register a request handler. The handler builds each task:
 
 ```ruby
-task = PatientHttp::RequestTask.new(
-  request: request,
-  task_handler: handler,
-  callback: "MyCallback"
-)
+PatientHttp.register_handler do |request:, callback:, callback_args: nil, raise_error_responses: nil|
+  task = PatientHttp::RequestTask.new(
+    request: request,
+    task_handler: MyTaskHandler.new(MyJobSystem.current_job_id),
+    callback: callback,
+    callback_args: callback_args,
+    raise_error_responses: raise_error_responses
+  )
+  processor.enqueue(task)
+  task.id
+end
+```
+
+To raise an error if a handler is already registered, use `PatientHttp.register_handler!`. To check whether a handler is registered, call `PatientHttp.handler_registered?`.
+
+To make `PatientHttp.configure` yield your own configuration class, register the integration as the configuration provider. The provider must respond to `new_configuration`, which returns a new configuration, and to `configure`:
+
+```ruby
+PatientHttp.register_configuration_provider(MyIntegration)
+```
+
+### Test an integration
+
+To test integration code without the async processor, use `SynchronousExecutor`. It runs a request on the calling thread, runs the optional hooks, and then calls the callback service:
 
+```ruby
 executor = PatientHttp::SynchronousExecutor.new(
   task,
   config: config,
@@ -836,25 +858,23 @@ executor = PatientHttp::SynchronousExecutor.new(
 executor.call
 ```
 
-## Integration
-
-For Sidekiq integration, see the [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) gem which provides workers, lifecycle hooks, crash recovery, and a Web UI built on this library.
-
-For Solid Queue integration, see the [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) gem which provides similar functionality for Solid Queue.
-
-When using an integration gem, you can use the [standard interface](#standard-interface) to make requests without coupling your code to the underlying processor or task handler implementations.
+## Installation
 
-For large language model (LLM) requests, see the [patient_llm](https://github.com/bdurand/patient_llm) gem which provides an integration for making LLM requests asynchronously via a variety of protocols.
+Most applications need only the integration gem for their job system, which depends on this gem:
 
-## Installation
+```ruby
+gem "patient_http-sidekiq"
+# or
+gem "patient_http-solid_queue"
+```
 
-Add this line to your application's Gemfile:
+To use this gem on its own, add it to your Gemfile:
 
 ```ruby
 gem "patient_http"
 ```
 
-Then execute:
+Then install it:
 
 ```bash
 bundle install
@@ -864,11 +884,17 @@ bundle install
 
 Open a pull request on [GitHub](https://github.com/bdurand/patient_http).
 
-Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
+Follow the [standardrb](https://github.com/testdouble/standard) style, and run `standardrb --fix` before you submit a pull request.
+
+Run the tests:
+
+```bash
+bundle exec rspec
+```
 
-The [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) and [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) gems each provide a test application for integration testing.
+The [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) and [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) gems each have a test app for integration testing.
 
-## Further Reading
+## Further reading
 
 - [Architecture](ARCHITECTURE.md)