butler-http 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6ecf83403f9b02b0164c16210c728e5fe96bd071bdd35a498d481c8956e5d63c
-  data.tar.gz: 1624f87cf1296c6b641ed2594875ce0eaba8b2d80e81fe5b2065995528a270b3
+  metadata.gz: d18e43e5caae734bc0c8acbea07465f8711561bf805508a5938c3e41ce51dec0
+  data.tar.gz: 8dbebb55192cf5064bd511d410075cc55753cf73184690bd04ffa2cd66e431ea
 SHA512:
-  metadata.gz: 0e052d6b126eec0b91f80b27c6a9c7a0897b78c5c513851c80e24021de5c70ba3d7efdd0e69ceb4f3c2d79a87a3455a2b21bc4480b2a5b35e63b49319828ac57
-  data.tar.gz: 628766bc50e91954b426f0d029d4babdb5083e2c2dfac629eee76be2dc824d7989f17e2b2808913ae7f8cc9410b7a47995129d23920c27cd6ed6ec703bf8d0dd
+  metadata.gz: 4e4873585f752cb1982436aecf495ed837ab4ff753cf9f591bce3bf820cfd9bba9e545d80cd387a3fe671954926db5bc7dd93738855f87aac0c2ee37287a71be
+  data.tar.gz: 1852f5d0644b66ecce66d51f8bb0ed42386b94f8689dc551da03c9031ce621bac7c79673114b3abbedce377675317d1c24621093c25f29cf648fe2233be60e3d

data/CHANGELOG.md

@@ -4,6 +4,209 @@ All notable changes to this project are documented here. Format loosely follows
 [Keep a Changelog](https://keepachangelog.com/); this project doesn't yet follow strict semver
 (pre-1.0 — see [README's "What's not here yet"](README.md#whats-not-here-yet)).
 
+## 0.2.0
+
+### Added
+
+- **`elastic-transport` integration** (`lib/butler/elasticsearch.rb`,
+  opt-in — `require "butler/elasticsearch"` after `elastic-transport`/
+  `elasticsearch` is already loaded). `Elastic::Transport::Transport::HTTP::Butler`
+  implements elastic-transport's own transport interface — the same one
+  `Transport::HTTP::Faraday`/`Transport::HTTP::Manticore` implement, not a
+  new extension point — so an `Elasticsearch::Client` can run entirely on
+  Butler:
+  ```ruby
+  Elasticsearch::Client.new(hosts: [...], transport_class: Elastic::Transport::Transport::HTTP::Butler)
+  ```
+  One `Butler::Client` per ES node, mirroring how the Faraday transport
+  builds one Faraday connection per node. Retry behavior is
+  host-count-aware: with more than one configured host, elastic-transport's
+  own cross-node failover is left in charge (Butler's own retry is set to
+  `max_attempts: 1`, avoiding double-retrying underneath that failover and
+  avoiding `RetryExhausted`, which elastic-transport doesn't recognize as
+  retryable). With exactly one configured host — no other node to fail
+  over to — `retry_on_failure`/`delay_on_retry` are translated into
+  Butler's own `retry:` config instead, adding real exponential
+  backoff+jitter in place of elastic-transport's own zero-backoff retry
+  loop, at the same attempt count as before. `host_unreachable_exceptions`
+  is mapped to Butler's own connection-level error classes so
+  dead-node detection/failover keeps working correctly. Covered by
+  `test/butler/elasticsearch_test.rb`, exercised through the real
+  `Elastic::Transport::Client` against a `FakeServer` standing in for an ES
+  node, not just the adapter class in isolation.
+
+  One caveat carried over from elastic-transport itself, not introduced
+  here: it declares `faraday` as a runtime dependency regardless of
+  `transport_class` (`Connections::Connection#full_url` uses
+  `Faraday::Utils::ParamsHash` for query-string encoding), so Faraday
+  stays in the dependency tree — it just performs no network I/O once
+  Butler is selected as the transport.
+
+- **`security.ca_file` / `security.ca_path` / `security.client_cert` /
+  `security.client_key`** for a private/custom CA or mutual-TLS client
+  certificates — previously the only TLS knob was the all-or-nothing
+  `security.verify_tls`. `client_cert`/`client_key` accept an
+  already-built `OpenSSL::X509::Certificate`/`OpenSSL::PKey::PKey`, a PEM
+  string, or a file path.
+
+- **`pool.max_connections_per_host`** (`nil`/unbounded by default,
+  matching prior behavior) to cap concurrent connections to a single
+  origin, using `async-pool`'s own `limit:` option. Bounds how many
+  connections a retry storm or a burst of concurrent `client.async` calls
+  can open against one struggling upstream. Included in `ConnectionPool`'s
+  cache key alongside the other connection-shaping settings (`proxy`,
+  `http_version`, `verify_tls`).
+
+- **`pool.max_connection_age`** (`nil`/disabled by default) to recycle a
+  connection after a configurable age, regardless of how actively it's
+  being used. `idle_timeout` only catches a connection gone stale from
+  disuse; a connection kept continuously busy never ages out through it.
+  Against a host whose public endpoint is a load balancer or proxy in
+  front of several backends — true of most managed/cloud services,
+  including Elastic Cloud — that could otherwise pin one long-lived
+  connection to a single backend indefinitely. Verified separately that
+  this isn't needed for connection-death recovery: a connection that
+  actually dies mid-use is already detected and rebuilt automatically by
+  `async-http`'s own pool.
+
+- **An `at_exit` hook** so a process exits normally after a synchronous
+  Butler call, without needing to be force-killed. `BackgroundReactor`'s
+  thread runs for the life of the process by design, with no exit
+  condition of its own; Ruby doesn't end a process while a non-daemon
+  thread is still alive, and this gem's target Ruby has removed
+  `Thread#daemon=`/`#daemon?` with no replacement. This hook reliably
+  covers the thread idling on its ordinary wait — i.e. every normal
+  shutdown — though, as tested directly, it can't recover a thread already
+  blocked in the Fiber scheduler's own I/O wait on a connection a bug left
+  open; that's the responsibility of closing connections correctly (see
+  the upload-streaming fix below), not a substitute for it.
+
+- A **Butler vs. Faraday benchmark suite for the Elasticsearch adapter**
+  (`benchmarks/elasticsearch_comparison.rb`), covering throughput,
+  latency percentiles, and TCP connection counts under concurrency.
+
+### Fixed
+
+- **`stream: true` (requesting a streamed response) raised `NoMethodError`
+  on every call that used it.** `Body.wrap` treated any `:stream` key as
+  "wrap this value as the request body," but `stream: true` is the
+  documented, separate flag for requesting a streamed *response*, carrying
+  no request body at all. The two options collided: `stream: true` was
+  always also read as an upload stream, building `StreamBody.new(true)`,
+  which raised as soon as anything tried to send it. Found while adding
+  the streaming test coverage below (there was none before this). Fixed
+  by only treating `:stream` as a request body when its value isn't
+  literally `true`; `stream: <enumerable>` for upload streaming is
+  unaffected.
+
+- **Upload streaming (`stream: <enumerable>`) leaked its connection,
+  which could prevent a process from exiting normally.** The same key
+  collision above had a second half: `Request.build` read the enumerable
+  itself as a truthy `stream:` value, silently also marking the request
+  as wanting a streamed response. Nothing ever consumed or closed that
+  response, so its connection was never released. Confirmed end to end —
+  reliably reproduced (20/20 runs) before the fix, 0/20 after — and fixed
+  by applying the same `stream: true`-only disambiguation to
+  `Request#stream?`.
+
+- **A per-attempt timeout covered dispatching the request, but not
+  reading the response body.** `Transport::Async.call` wrapped the
+  initial dispatch in `Resilience::Timeout.enforce`, but the body read
+  (`Security::Limits.read_body`) happened afterward, outside that
+  timeout. A server that sent headers promptly but stalled or drip-fed
+  the body could hang regardless of `deadline:`/`read_timeout`/
+  `request_timeout`. The full attempt — dispatch and body read — is now
+  wrapped in one timeout; the `stream: true` case is unaffected, since
+  actual chunk-by-chunk consumption is documented to happen outside the
+  attempt timeout, paced by the caller.
+
+- **A retryable response cut short by deadline expiry was returned as-is
+  instead of raising `Butler::Errors::TimeoutError`.**
+  `RetryPolicy#retry_on_response?` treated "not retryable," "max attempts
+  reached," and "deadline expired" the same way, but only the deadline
+  case should surface as a timeout rather than a normal result. The
+  equivalent exception-retry path already handled this correctly; a new
+  `RetryPolicy#retryable_response?` (status/method eligibility only, no
+  deadline or attempt-count check) lets `RetryMiddleware` distinguish
+  "nothing left to retry" from "this would have been retried, but time
+  ran out."
+
+- **`ConnectionPool`'s cache key omitted `security.ca_file`/`ca_path`/
+  `client_cert`/`client_key`.** Two configurations differing only in TLS
+  identity — for example, different mutual-TLS client certificates —
+  could hash to the same pool entry, so the second configuration's
+  requests would reuse a connection authenticated as the first.
+  Reproduced directly against `ConnectionPool#acquire` before fixing.
+  `client_cert`/`client_key` are fingerprinted by `#object_id` rather
+  than content, both to avoid re-parsing cost and so the fingerprint
+  never holds actual key material.
+
+- **The `elastic-transport` integration raised `NoMethodError` against
+  elastic-transport versions predating OpenTelemetry support** — confirmed
+  against the published 8.1.0 release, which has no
+  `capture_otel_span_attributes` method on `Transport::Base` at all (added
+  later, present from 8.5.x). `perform_request` called it unconditionally,
+  mirroring `Transport::HTTP::Faraday`'s newer shape without its implicit
+  version guard. Found by diffing 8.1.0 against 8.5.3 before relying on
+  this integration against an older elastic-transport release, and
+  verified fixed by loading the real 8.1.0 gem directly and issuing
+  requests through it. Now only called when it actually exists.
+
+- **`config.proxy` was accepted but never used** — `Transport::Async.build_endpoint`
+  never read it, so requests went straight to the target regardless of a
+  configured proxy. Now tunnels through the configured proxy via a real
+  HTTP CONNECT (`Async::HTTP::Proxy`), for both HTTP and HTTPS targets;
+  `http://user:pass@host:port` proxy URLs send credentials as
+  `Proxy-Authorization: Basic ...`. A failed CONNECT is classified as
+  `Butler::Errors::ConnectionError`, keeping `rescue Butler::Errors::Error`
+  a safe catch-all. Covered by `test/butler/transport_proxy_test.rb`,
+  which runs real traffic through a CONNECT-tunneling fake proxy.
+
+- **A half-open circuit-breaker probe cancelled via task cancellation
+  could leave that host's circuit open permanently.** `CircuitBreaker#call`
+  only cleared its `probing` flag on `StandardError`; `Async::Stop` (raised
+  when a task is cancelled, e.g. by `Client#async`'s own cleanup after a
+  sibling task raised) is an `Exception`, not a `StandardError`, so it
+  bypassed that entirely, leaving every subsequent request to that host
+  raising `CircuitOpen` until the process restarted. `probing` now clears
+  on any non-`StandardError` exception too, without counting it as a
+  tracked failure.
+
+- **An oversized response leaked its connection.**
+  `Security::Limits.check_headers!`/`.read_body` can raise
+  `LimitExceeded` before anything takes ownership of closing the
+  underlying response; `build_response` didn't account for that, so a
+  response tripping `max_header_size`/`max_response_size` leaked its
+  connection instead of returning it to the pool. `raw_response` is now
+  closed on any error raised before a `Response`/`Stream` exists to take
+  over that responsibility.
+
+- **A relative path with no `base_url` configured raised a bare
+  `URI::InvalidURIError`** instead of a `Butler::Errors::Error`, breaking
+  the documented guarantee that `rescue Butler::Errors::Error` is a safe
+  catch-all. This and other unparseable URLs now raise
+  `Butler::Errors::ConfigurationError` with a message naming the actual
+  problem.
+
+### Changed
+
+- **`circuit_breaker.enabled` now defaults to `false`.** Opt in explicitly
+  (`Butler::Client.new(circuit_breaker: { enabled: true })`, or globally
+  via `Butler.configure`) — a breaker tripping unexpectedly for a caller
+  who never requested one is a worse surprise than not having one by
+  default. This is a default-behavior change: anyone relying on it being
+  enabled by default needs to opt in explicitly after upgrading.
+
+### Verified
+
+- **Extensive concurrency stress-testing against a real server** — threads
+  and `client.async` Fibers, up to 2000 concurrent, connection pool sizes
+  down to a single shared connection, mixed GET/POST with randomized
+  response/body sizes, and a real mid-body cancellation via `deadline:`.
+  Zero response/request mixing found across tens of thousands of
+  requests; `async-http` correctly isolates concurrent requests even
+  under maximal connection reuse.
+
 ## 0.1.1
 
 - Added `lib/butler/http.rb` (requiring `lib/butler.rb`) so `gem "butler-http"` works with Bundler's

data/README.md

@@ -47,6 +47,7 @@ end
   - [Middleware](#middleware)
   - [Testing — no WebMock/VCR needed](#testing--no-webmockvcr-needed)
   - [Rails](#rails)
+  - [Elasticsearch](#elasticsearch)
   - [Errors](#errors)
 - [Configuration reference](#configuration-reference)
 - [Benchmarks](#benchmarks)
@@ -380,6 +381,8 @@ client.put("/orders/42", json: { status: "shipped" }, idempotent: true) # opt in
 
 ### Circuit breaker
 
+Off by default — opt in per client (or globally via `Butler.configure`):
+
 ```ruby
 client = Butler::Client.new(
   circuit_breaker: { enabled: true, scope: :host, failure_threshold: 5, recovery_timeout: 30 },
@@ -406,13 +409,20 @@ The exact rules for what counts, since they matter in production:
   429) never counts as a circuit-breaker failure this way — Butler
   doesn't raise on error responses unless `raise_on_error: true`, and the
   breaker's `failure:` check is specifically `response.server_error?`.
-- Any **raised exception** always counts as a failure, regardless of
-  class — `ConnectionError`, `TimeoutError`, `TLSError` and its
-  `CertificateVerificationError` subclass, `DNSFailure`,
-  `ProtocolError`, `RetryExhausted`, all of it.
-- `Butler::Errors::CircuitOpen` itself is the one exception excluded —
-  a short-circuited call (the breaker already open) never counts toward
-  its own failure count.
+- Any **raised `Butler::Errors::Error`** (or any other `StandardError`)
+  always counts as a failure, regardless of class — `ConnectionError`,
+  `TimeoutError`, `TLSError` and its `CertificateVerificationError`
+  subclass, `DNSFailure`, `ProtocolError`, `RetryExhausted`, all of it.
+- `Butler::Errors::CircuitOpen` itself is excluded — a short-circuited
+  call (the breaker already open) never counts toward its own failure
+  count.
+- A non-`StandardError` signal cutting a call short — chiefly
+  `Async::Stop` from task cancellation (e.g. a sibling task in the same
+  `client.async` block raising) — isn't counted as a failure either,
+  since it isn't evidence the *host* failed. A half-open probe cut short
+  this way still clears its own in-flight state so the next request can
+  probe again, rather than leaving the circuit wedged open until the
+  process restarts.
 
 ### Security
 
@@ -519,6 +529,44 @@ a service-object pattern, subscribing to `"butler.request"` correctly (and
 a real subtlety around *where* the event's duration actually lives) — in
 [docs/rails.md](docs/rails.md).
 
+### Elasticsearch
+
+Opt-in (not loaded by `require "butler"`) — swaps the official
+`elasticsearch`/`elastic-transport` client's HTTP engine from Faraday to
+Butler, using the same `transport_class:` extension point that gem's own
+`Transport::HTTP::Manticore`/`Transport::HTTP::Curb` are built on:
+
+```ruby
+require "butler"
+require "elasticsearch"          # or just "elastic-transport"
+require "butler/elasticsearch"
+
+client = Elasticsearch::Client.new(
+  hosts: ["https://es.example.com:9200"],
+  transport_class: Elastic::Transport::Transport::HTTP::Butler,
+)
+
+client.search(index: "my-index", body: { query: { match_all: {} } })
+```
+
+One `Butler::Client` per node — each keeps its own pooled connections,
+independent of every other node in the cluster — with retries forced off
+(`max_attempts: 1`): elastic-transport's own `Base#perform_request` already
+retries and fails over across every node itself, so Butler only needs to
+report a failed node's connection error back cleanly, not retry underneath
+that layer too. `transport_options`/`ssl` (timeouts, a custom CA,
+mutual-TLS client cert, extra headers — see
+[Configuration reference](#configuration-reference)'s `security.*`
+options) map onto the per-node `Butler::Client`'s config.
+
+One dependency caveat carried over from elastic-transport itself, not
+introduced by this integration: it declares `faraday` as a runtime
+dependency regardless of `transport_class` (query-string encoding in its
+`Connection#full_url` uses `Faraday::Utils::ParamsHash`), so Faraday stays
+in the dependency tree — it just does zero network I/O once Butler is
+selected. Every actual connection, retry, and TLS handshake goes through
+Butler.
+
 ### Errors
 
 Every error Butler raises descends from `Butler::Errors::Error`:
@@ -578,7 +626,7 @@ end
 | `retry.retryable_status_codes` | `[408,425,429,500,502,503,504]` | |
 | `retry.base_delay` / `retry.max_delay` | `0.1` / `5.0` | Seconds |
 | `retry.jitter` | `true` | Equal-jitter (delay scaled by a random factor in `[0.5, 1.0)`) |
-| `circuit_breaker.enabled` | `true` | |
+| `circuit_breaker.enabled` | `false` | Opt-in — see [Circuit breaker](#circuit-breaker) |
 | `circuit_breaker.scope` | `:host` | or `:client`, to share one breaker across every host |
 | `circuit_breaker.failure_threshold` | `5` | |
 | `circuit_breaker.recovery_timeout` | `30` | Seconds before a half-open probe is allowed |
@@ -590,6 +638,8 @@ end
 | `security.strip_credentials_on_redirect` | `true` | |
 | `pool.max_connections` | `100` | Distinct origins kept warm at once (LRU-evicted past this) |
 | `pool.idle_timeout` | `60` | Seconds a pooled connection may sit unused before it's rebuilt rather than reused |
+| `pool.max_connections_per_host` | `nil` (unbounded) | Caps concurrent connections to one origin — a retry storm or a burst of concurrent `client.async` calls against one struggling upstream otherwise has nothing stopping it from opening one connection per in-flight request |
+| `pool.max_connection_age` | `nil` (no forced recycling) | Forces a connection to close and rebuild once it's been open this many seconds, however actively it's being used — unlike `idle_timeout` (which only catches a connection gone stale from *disuse*), this bounds how long a *busy* connection stays pinned to whichever single backend it first connected to behind a load balancer/proxy (true of most managed/cloud services) |
 | `telemetry.enabled` | `true` | |
 | `telemetry.opentelemetry` | `:auto` | Spans are emitted automatically whenever `opentelemetry-api` is already loaded |
 | `telemetry.logger` | `nil` (falls back to `Logger.new($stdout)`, or `Rails.logger` under Rails) | |

data/lib/butler/async.rb

@@ -83,6 +83,18 @@ class Butler
         @mutex.synchronize do
           @thread = nil
           @pid = nil
+          # A forked child inherits the parent's at_exit hooks (and
+          # @exit_hook_registered's already-true value) along with
+          # everything else in its copied memory — but the Thread object
+          # that hook's closure captured belongs to the *parent* and
+          # doesn't exist in this process (threads don't survive fork).
+          # Without resetting this too, the next ensure_running! in this
+          # child spawns its own new reactor thread but skips registering
+          # an at_exit for *that* one, since the flag already reads true —
+          # leaving exactly the forked-worker case this whole mechanism
+          # (and Butler.reset_connections!, which calls this on every
+          # Puma/preload_app! fork) exists for uncovered.
+          @exit_hook_registered = false
         end
       end
 
@@ -96,9 +108,45 @@ class Butler
 
           @pid = ::Process.pid
           @thread = ::Thread.new { run }
+          ensure_exit_hook_registered!
         end
       end
 
+      # This thread runs `loop { @jobs.pop; ... }` forever by design (so
+      # it's ready for the *next* bare call without respawning), with no
+      # exit condition of its own — so without this, a process that made
+      # even one bare synchronous Butler call would never exit on its own
+      # once the main thread/script finished, full stop, leak or no leak.
+      # Ruby removed Thread#daemon=/#daemon? (no replacement — confirmed
+      # against this gem's actual target Ruby, which has neither), so an
+      # at_exit hook killing this thread is the mechanism available.
+      #
+      # Scope, confirmed empirically rather than assumed: Thread#kill
+      # reliably ends this thread while it's idling on the ordinary
+      # `@jobs.pop` wait — covers every normal shutdown. It does *not*
+      # reliably end it if the thread is instead blocked inside the Fiber
+      # scheduler's own I/O wait on a connection some bug left open and
+      # unclosed (confirmed with the exact connection-leak bug fixed
+      # elsewhere in this version, before that fix existed: `kill` plus an
+      # explicit `join` left the thread still alive). This hook is real
+      # insurance against the ordinary case, not a substitute for closing
+      # connections correctly — a genuine future leak can still wedge a
+      # process open despite it. Abandoning any request this thread is
+      # still mid-flight on when the process actually exits is an
+      # acceptable, expected cost — any HTTP client loses in-flight
+      # work if its process is killed — and is categorically better than a
+      # client-library bug silently preventing a script, rake task, or
+      # test run from ever exiting on its own. Registered at most once
+      # per process (guarded by @exit_hook_registered, itself only ever
+      # read/written inside the @mutex-held section above/here).
+      def ensure_exit_hook_registered!
+        return if @exit_hook_registered
+
+        @exit_hook_registered = true
+        thread = @thread
+        at_exit { thread.kill }
+      end
+
       def running?
         @thread&.alive? && @pid == ::Process.pid
       end

data/lib/butler/body.rb

@@ -9,7 +9,19 @@ module Butler::Body
       JSONBody.new(options[:json])
     elsif options.key?(:form)
       StringBody.new(::URI.encode_www_form(options[:form]), content_type: "application/x-www-form-urlencoded")
-    elsif options.key?(:stream)
+    elsif options.key?(:stream) && options[:stream] != true
+      # The `stream:` key is overloaded: `stream: true` (a bare boolean)
+      # means "give me back a streamed *response*" — handled entirely
+      # separately, by Request#stream? / Transport::Async#build_response —
+      # and carries no request body at all. `stream: <enumerable>` means
+      # "stream *this* as the request body" (uploads). `!= true` is what
+      # tells them apart: nothing meant as an upload stream is ever
+      # literally the boolean true. Before this check, `stream: true` for
+      # a streamed *response* (the common, README-documented case) was
+      # always misread as an upload stream too, building
+      # StreamBody.new(true) — which blew up the moment anything tried to
+      # actually send it (`true` doesn't respond to #each) on *every*
+      # streamed-response request, GET included.
       StreamBody.new(options[:stream])
     elsif options.key?(:io)
       IOBody.new(options[:io])

data/lib/butler/configuration.rb

@@ -25,7 +25,7 @@ class Butler::Configuration
     attr_accessor :enabled, :scope, :failure_threshold, :recovery_timeout, :max_tracked_hosts
 
     def initialize
-      @enabled = true
+      @enabled = false
       @scope = :host # or :client, to share one breaker across every origin
       @failure_threshold = 5
       @recovery_timeout = 30
@@ -35,7 +35,8 @@ class Butler::Configuration
 
   class SecurityOptions
     attr_accessor :verify_tls, :allowed_hosts, :blocked_hosts,
-                  :max_response_size, :max_header_size, :strip_credentials_on_redirect
+                  :max_response_size, :max_header_size, :strip_credentials_on_redirect,
+                  :ca_file, :ca_path, :client_cert, :client_key
 
     def initialize
       @verify_tls = true
@@ -44,6 +45,19 @@ class Butler::Configuration
       @max_response_size = 50 * 1024 * 1024
       @max_header_size = 64 * 1024
       @strip_credentials_on_redirect = true
+      # A private/custom CA (e.g. a self-managed cluster's generated root
+      # cert — common for Elasticsearch, internal services, etc.) — nil
+      # (default) verifies against the system trust store only, same as
+      # before these existed. ca_file: a single PEM bundle; ca_path: a
+      # directory of hashed cert files (OpenSSL::SSL::SSLContext#ca_path
+      # semantics) — either or both may be set.
+      @ca_file = nil
+      @ca_path = nil
+      # Mutual TLS: an OpenSSL::X509::Certificate (or a PEM string/path —
+      # see Security::TLS.context_for) and matching OpenSSL::PKey private
+      # key. Both must be set together or neither has any effect.
+      @client_cert = nil
+      @client_key = nil
     end
 
     def initialize_copy(source)
@@ -54,11 +68,33 @@ class Butler::Configuration
   end
 
   class PoolOptions
-    attr_accessor :max_connections, :idle_timeout
+    attr_accessor :max_connections, :idle_timeout, :max_connections_per_host, :max_connection_age
 
     def initialize
       @max_connections = 100
       @idle_timeout = 60
+      # nil (default) matches async-pool's own default: unbounded concurrent
+      # connections to one origin. Set to cap how many sockets Butler will
+      # ever hold open to a single host at once — a retry storm or a burst
+      # of concurrent `client.async` calls against one struggling upstream
+      # otherwise has nothing stopping it from opening one connection per
+      # in-flight request.
+      @max_connections_per_host = nil
+      # nil (default, no forced recycling) preserves the original
+      # behavior: a healthy, actively-used connection is kept forever,
+      # however long that is. A *busy* connection never ages out via
+      # idle_timeout (it's never idle), so against a host whose public
+      # endpoint is actually a load balancer or proxy in front of several
+      # backends (true of most managed/cloud services — Elastic Cloud
+      # included), one long-lived pooled connection stays pinned to
+      # whichever single backend it first connected to for as long as it
+      # keeps getting used, however unevenly that distributes load. Set
+      # this to force a connection to close and rebuild (fresh DNS
+      # resolution, a fresh chance at the LB's own distribution) once it's
+      # been open this many seconds, regardless of how actively it's being
+      # used — a pure availability/distribution knob, unrelated to
+      # idle_timeout's "has this connection gone stale from disuse" job.
+      @max_connection_age = nil
     end
   end
 

data/lib/butler/connection.rb

@@ -7,15 +7,25 @@
 class Butler::Connection
   attr_reader :origin_key, :async_client, :http_version
 
-  def initialize(origin_key:, async_client:, http_version:)
+  def initialize(origin_key:, async_client:, http_version:, proxy_client: nil)
     @origin_key = origin_key
     @async_client = async_client
     @http_version = http_version
+    @proxy_client = proxy_client
   end
 
   def close
     @async_client&.close
   rescue StandardError
     nil
+  ensure
+    # Only set when config.proxy routed this connection through a CONNECT
+    # tunnel (see Transport::Async.apply_proxy) — the client holding the
+    # tunnel to the proxy server itself, distinct from @async_client above.
+    begin
+      @proxy_client&.close
+    rescue StandardError
+      nil
+    end
   end
 end

data/lib/butler/connection_pool.rb

@@ -6,7 +6,7 @@
 # reuse the same underlying Async::HTTP::Client instead of rebuilding one
 # (and paying DNS/TCP/TLS setup again) on every call.
 class Butler::ConnectionPool
-  Entry = Struct.new(:connection, :last_used_at)
+  Entry = Struct.new(:connection, :last_used_at, :created_at)
 
   def initialize(config)
     @config = config
@@ -20,17 +20,20 @@ class Butler::ConnectionPool
 
     @mutex.synchronize do
       if (entry = @entries.delete(key))
-        if now - entry.last_used_at <= config.pool.idle_timeout
+        max_age = config.pool.max_connection_age
+        if now - entry.last_used_at > config.pool.idle_timeout
+          entry.connection.close # sat idle past pool.idle_timeout — rebuild fresh below rather than reuse
+        elsif max_age && now - entry.created_at > max_age
+          entry.connection.close # past pool.max_connection_age, however actively it's been used — see Configuration
+        else
           entry.last_used_at = now
           @entries[key] = entry # touch: move to the end
           return entry.connection
-        else
-          entry.connection.close # sat idle past pool.idle_timeout — rebuild fresh below rather than reuse
         end
       end
 
       connection = Butler::Transport.current.build_connection(uri, config)
-      @entries[key] = Entry.new(connection, now)
+      @entries[key] = Entry.new(connection, now, now)
       evict_if_needed!
       connection
     end
@@ -47,7 +50,23 @@ class Butler::ConnectionPool
 
   def fingerprint(uri, config)
     transport_name = Butler::Transport.current.equal?(Butler::Testing::FakeTransport) ? "fake" : "real"
-    "#{uri.origin_key}|proxy=#{config.proxy}|http_version=#{config.http_version}|verify_tls=#{config.security.verify_tls}|transport=#{transport_name}"
+    security = config.security
+    # client_cert/client_key by #object_id, never by content: the same
+    # Configuration's cert/key reference is reused for every call across a
+    # Client's lifetime (the realistic case, and the only one that matters
+    # for pooling today -- there's no per-call security override), so
+    # identity alone already distinguishes "same client" from "different
+    # client" correctly; embedding actual PEM/key content into this string
+    # would also risk leaking key material into anything that ever logs a
+    # fingerprint. Without ca_file/ca_path/client_cert/client_key here, two
+    # Configurations that differ *only* in TLS identity hashed to the same
+    # key, so the second call silently got back a connection built from the
+    # first's certificate/CA -- a real cross-identity mix-up, not just a
+    # missed-reuse inefficiency.
+    "#{uri.origin_key}|proxy=#{config.proxy}|http_version=#{config.http_version}|verify_tls=#{security.verify_tls}" \
+      "|ca_file=#{security.ca_file}|ca_path=#{security.ca_path}" \
+      "|client_cert=#{security.client_cert&.object_id}|client_key=#{security.client_key&.object_id}" \
+      "|pool_limit=#{config.pool.max_connections_per_host}|transport=#{transport_name}"
   end
 
   def evict_if_needed!

data/lib/butler/elasticsearch.rb

@@ -0,0 +1,205 @@
+# An elastic-transport HTTP transport backed by Butler instead of Faraday.
+# Not required by `require "butler"` — this is an opt-in integration for a
+# host that already depends on the `elastic-transport` gem (directly, or via
+# the `elasticsearch`/`elasticsearch-api` gems), loaded explicitly:
+#
+#   require "butler"
+#   require "elastic-transport"       # or "elasticsearch" — either defines Elastic::Transport
+#   require "butler/elasticsearch"
+#
+#   client = Elasticsearch::Client.new(
+#     hosts: ["https://es.example.com:9200"],
+#     transport_class: Elastic::Transport::Transport::HTTP::Butler,
+#   )
+#
+# elastic-transport already supports swapping its HTTP engine this way —
+# Transport::HTTP::Faraday is just its *default* transport_class, and it
+# ships two others built the same way this one is (Transport::HTTP::Curb,
+# Transport::HTTP::Manticore for JRuby) — this follows that exact pattern,
+# not a new extension point invented for Butler.
+#
+# One real constraint carried over from elastic-transport itself, not
+# introduced here: Transport::Connections::Connection#full_url always
+# builds query strings via Faraday::Utils::ParamsHash, and elastic-transport
+# declares `faraday` as one of its own runtime dependencies regardless of
+# transport_class — so Faraday stays in the dependency tree even with this
+# transport selected. It does zero network I/O at that point, though: the
+# actual connections, TLS, retries, and connection pooling for every
+# request all go through Butler once this file is loaded.
+raise LoadError, "butler/elasticsearch requires the elastic-transport gem (or elasticsearch/elasticsearch-api, which depend on it) to be loaded first" unless defined?(::Elastic::Transport::Transport::Base)
+
+require "base64"
+
+module Elastic
+  module Transport
+    module Transport
+      module HTTP
+        # @see Transport::Base
+        class Butler
+          include Elastic::Transport::Transport::Base
+
+          # @return [Elastic::Transport::Transport::Response]
+          # @see    Transport::Base#perform_request
+          def perform_request(method, path, params = {}, body = nil, headers = nil, opts = {})
+            super do |connection, url|
+              # capture_otel_span_attributes only exists on elastic-transport
+              # versions that added OpenTelemetry support (present in 8.5.x,
+              # absent in 8.1.x's Base module entirely — confirmed by diffing
+              # both) — calling it unconditionally would raise NoMethodError
+              # on literally every request against an older elastic-transport.
+              # Transport::HTTP::Faraday only gained this same call once that
+              # support landed too, so skipping it on older versions matches
+              # that transport's own behavior at the time, not a Butler-only
+              # gap.
+              capture_otel_span_attributes(connection, url) if respond_to?(:capture_otel_span_attributes, true)
+              body = body ? __convert_to_json(body) : nil
+              body, headers = compress_request(body, parse_headers(connection, headers))
+
+              # Base#perform_request already folded `params` into `url`
+              # (Connection#full_url) before handing it to this block, so
+              # it must not be passed again here — Butler would otherwise
+              # encode it a second time onto what's already a complete,
+              # absolute URL.
+              response = connection.connection.public_send(
+                http_method(method), url, body: body, headers: headers || {},
+              )
+              Response.new(response.status, decompress_response(response.body), response.headers)
+            end
+          end
+
+          # Builds and returns a connection: one Butler::Client per host,
+          # exactly like Transport::HTTP::Faraday builds one Faraday
+          # connection per host — each keeps its own pooled connections,
+          # retry/circuit-breaker state, independent of every other node.
+          #
+          # @return [Connections::Connection]
+          def __build_connection(host, options = {}, block = nil)
+            client = ::Butler::Client.new(**client_options_for(host, options))
+            Connections::Connection.new(host: host, connection: client)
+          end
+
+          # @return [Array]
+          def host_unreachable_exceptions
+            [
+              ::Butler::Errors::ConnectionError,
+              ::Butler::Errors::TimeoutError,
+              ::Butler::Errors::TLSError,
+              ::Butler::Errors::DNSFailure,
+              ::Butler::Errors::ProtocolError,
+            ]
+          end
+
+          private
+
+          def http_method(method)
+            name = method.to_s.downcase.to_sym
+            return name if ::Butler::Client::HTTP_METHODS.include?(name)
+
+            raise ArgumentError, "Method #{method} not supported"
+          end
+
+          # Per-request headers merged with whatever this connection's
+          # client already carries as default_headers (API-key/custom
+          # headers configured globally via transport_options above) —
+          # Butler itself also does per-request-overrides-default merging,
+          # but compress_request (called right after this) needs the final
+          # merged Content-Encoding decision up front.
+          def parse_headers(connection, headers)
+            defaults = connection.connection.configuration.default_headers
+            return defaults.to_h if headers.nil?
+            return headers if defaults.empty?
+
+            defaults.to_h.merge(headers)
+          end
+
+          def client_options_for(host, options)
+            transport_options = options[:transport_options] || {}
+            request_options = transport_options[:request] || {}
+            ssl_options = transport_options[:ssl] || options[:ssl] || {}
+
+            {
+              base_url: __full_url(host),
+              connect_timeout: request_options[:open_timeout],
+              read_timeout: request_options[:timeout],
+              write_timeout: request_options[:timeout],
+              headers: default_headers_for(host, transport_options),
+              security: security_options_for(ssl_options),
+              retry: retry_options_for(options),
+            }.compact
+          end
+
+          # With more than one configured host, elastic-transport's own
+          # Base#perform_request retry loop is what actually provides value:
+          # get_connection cycles to a *different* node on retry (see
+          # Connections::Collection/Selector + reload_on_failure), which a
+          # single Butler::Client has no way to do since it only ever talks
+          # to the one host it was built for in __build_connection above.
+          # Leaving Butler's own retry on too would compound underneath
+          # that — every Butler retry against this one node happening
+          # before elastic-transport ever got to try the next node — and
+          # wrap a connection failure as Butler::Errors::RetryExhausted,
+          # which isn't in host_unreachable_exceptions and so wouldn't be
+          # recognized as failover-worthy at all. So multi-host leaves
+          # cross-node failover entirely to elastic-transport.
+          #
+          # Exactly one configured host (e.g. a single-node ES install, or
+          # a load balancer URL in front of a real cluster) is a different
+          # story: there's no other node to fail over to, so
+          # retry_on_failure/delay_on_retry
+          # would just be retrying this same node — hand that fully to
+          # Butler instead, which adds real exponential backoff+jitter.
+          # elastic-transport's own retry loop has no backoff at all
+          # (delay_on_retry defaults to 0ms), so repeated *immediate*
+          # retries against an already-struggling single node can make
+          # things worse, not better. retry_on_failure's own count is
+          # preserved (true => its own DEFAULT_MAX_RETRIES of 3) so
+          # switching to Butler doesn't silently change how many attempts
+          # a request gets by default — only delay_on_retry, when left
+          # unset (0, same default), defers to Butler's own base_delay
+          # (0.1s) instead of literally meaning "no delay," since a literal
+          # 0 is the bug being fixed here, not a setting to preserve.
+          def retry_options_for(options)
+            return { max_attempts: 1 } if hosts.size > 1
+
+            retry_on_failure = options[:retry_on_failure]
+            retries =
+              case retry_on_failure
+              when true then Elastic::Transport::Transport::Base::DEFAULT_MAX_RETRIES
+              when Integer then retry_on_failure
+              else 0
+              end
+
+            retry_config = { max_attempts: retries + 1, jitter: true }
+            delay_on_retry = options[:delay_on_retry].to_i
+            retry_config[:base_delay] = delay_on_retry / 1000.0 if delay_on_retry.positive?
+            retry_config
+          end
+
+          def default_headers_for(host, transport_options)
+            headers = (transport_options[:headers] || {}).dup
+            headers["Content-Type"] ||= "application/json"
+            headers["Authorization"] ||= basic_auth_header(host) if host[:user]
+            headers
+          end
+
+          def basic_auth_header(host)
+            "Basic #{::Base64.strict_encode64("#{host[:user]}:#{host[:password]}")}"
+          end
+
+          # Maps elastic-transport's own `ssl:` option shape
+          # (verify/ca_file/ca_path/client_cert/client_key — see
+          # elastic-transport's README) onto Butler::Configuration::SecurityOptions.
+          def security_options_for(ssl_options)
+            {
+              verify_tls: ssl_options.key?(:verify) ? !!ssl_options[:verify] : nil,
+              ca_file: ssl_options[:ca_file],
+              ca_path: ssl_options[:ca_path],
+              client_cert: ssl_options[:client_cert],
+              client_key: ssl_options[:client_key],
+            }.compact
+          end
+        end
+      end
+    end
+  end
+end

data/lib/butler/headers.rb

@@ -80,7 +80,14 @@ class Butler::Headers
   # per-request headers override client-level defaults, rather than
   # duplicating them).
   def merge(other)
-    merged = Butler::Headers.new(@pairs)
+    # `dup` (→ #initialize_copy's `@pairs.map(&:dup)`) rather than
+    # `Butler::Headers.new(@pairs)`: every pair here is already
+    # downcased/stringified, so re-running `.new`'s own `add` (which
+    # redoes both of those per pair) on them is pure waste — this runs on
+    # every single request (Request.build merges config.default_headers
+    # with the per-call headers:, every time, even when there's nothing to
+    # actually merge in).
+    merged = dup
     Butler::Headers.coerce(other).each { |k, v| merged[k] = v }
     merged
   end

data/lib/butler/pipeline/middlewares/retry_middleware.rb

@@ -35,6 +35,24 @@ class Butler
               next
             end
 
+            # retry_on_response? above folds three different reasons into
+            # one false: this response was never retryable to begin with
+            # (a 200, a non-retryable 4xx — return it, nothing happened),
+            # max_attempts was reached (a real RetryExhausted-shaped
+            # outcome, but response-based retries don't raise on exhaustion
+            # the way the exception path does — returning the last response
+            # is the documented behavior there), or the *deadline* ran out
+            # before a response that genuinely would have been retried
+            # could get another attempt. Only that last case must not
+            # silently hand back a stale response as if the call completed
+            # normally within its budget — the whole point of deadline: is
+            # a caller can rely on either a real response *or*
+            # Butler::Errors::TimeoutError, never a response that arrived
+            # after they were told to stop waiting.
+            if context.deadline.expired? && @retry_policy.retryable_response?(request: context.request, response: response, retry_options: retry_options)
+              raise Butler::Errors::TimeoutError, "deadline exceeded after #{attempt} attempt(s), last response was #{response.status}"
+            end
+
             return response
           end
         end

data/lib/butler/request.rb

@@ -21,7 +21,19 @@ class Butler::Request
       body: body,
       idempotent: options.fetch(:idempotent, false),
       basic_auth: options[:basic_auth],
-      stream: options.fetch(:stream, false),
+      # The same collision Body.wrap guards against (see its comment):
+      # `stream:` means "give me a streamed response" only when it's
+      # literally `true`. `stream: <enumerable>` (upload streaming) must
+      # NOT also set this — options.fetch(:stream, false) used to return
+      # the enumerable itself (truthy) in that case, silently making
+      # Transport::Async#build_response wrap the *response* in a
+      # Butler::Stream too, one nobody asked for and nobody consumes or
+      # closes. An abandoned Butler::Stream never closes its underlying
+      # connection (that only happens via #each_chunk/#close), so every
+      # upload-streamed call leaked its connection — real enough to wedge
+      # a whole process open indefinitely rather than exit cleanly, since
+      # Butler's background reactor thread has live I/O to keep watching.
+      stream: options[:stream] == true,
     )
   end
 

data/lib/butler/resilience/circuit_breaker.rb

@@ -49,11 +49,28 @@ class Butler
         rescue StandardError
           record_failure!(state, circuit_breaker_options)
           raise
+        rescue Exception # rubocop:disable Lint/RescueException
+          # A non-StandardError signal cutting the attempt short — chiefly
+          # Async::Stop, raised into a task when something cancels it (e.g.
+          # Client#async's `ensure` stopping sibling tasks after one of them
+          # raised). Not evidence the *host* itself is failing, so this
+          # shouldn't move failure_count/re-open the circuit the way a real
+          # StandardError does — but `probing` must still clear no matter
+          # how the attempt ends, or a half-open probe cut short this way
+          # leaves it `true` forever, wedging every future request to this
+          # host on "circuit half-open probe already in flight" with no way
+          # to recover short of restarting the process.
+          clear_probing!(state)
+          raise
         end
       end
 
       private
 
+      def clear_probing!(state)
+        @mutex.synchronize { state.probing = false }
+      end
+
       def fetch(scope_key, circuit_breaker_options)
         @mutex.synchronize do
           state = @states.delete(scope_key) || HostState.new(:closed, 0, nil, false)

data/lib/butler/resilience/retry_policy.rb

@@ -45,6 +45,20 @@ class Butler
       def retry_on_response?(request:, response:, attempt:, deadline:, retry_options:)
         return false if deadline.expired?
         return false if attempt >= retry_options.max_attempts
+
+        retryable_response?(request: request, response: response, retry_options: retry_options)
+      end
+
+      # Whether +response+ would be retried on its own merits — status code
+      # + method/idempotency eligibility — ignoring the deadline and
+      # attempt-count budget entirely. Separated out from #retry_on_response?
+      # so RetryMiddleware can tell "this response was never going to be
+      # retried anyway" (a 200, or a 404 — just return it) apart from "this
+      # *would* have been retried, but the deadline ran out first" (the
+      # caller's deadline: budget was exhausted, which must surface as
+      # Butler::Errors::TimeoutError, not a stale response silently handed
+      # back as if the call completed normally within budget).
+      def retryable_response?(request:, response:, retry_options:)
         return false unless retry_options.retryable_status_codes.include?(response.status)
 
         request.retry_eligible?

data/lib/butler/security/tls.rb

@@ -8,17 +8,32 @@ class Butler
 
       def context_for(config)
         context = ::OpenSSL::SSL::SSLContext.new
-        if config.security.verify_tls
+        security = config.security
+
+        if security.verify_tls
           # #set_params (not a bare verify_mode= assignment) is what loads
           # the system's trusted CA store — a plain SSLContext.new has none
           # at all, so every certificate would fail verification regardless
           # of how legitimate it is.
-          context.set_params(verify_mode: ::OpenSSL::SSL::VERIFY_PEER)
+          params = { verify_mode: ::OpenSSL::SSL::VERIFY_PEER }
+          # A private/custom CA (e.g. a self-managed Elasticsearch cluster's
+          # generated root cert) *adds* to the system trust store set up by
+          # set_params above rather than replacing it — set_params applies
+          # ca_file/ca_path itself when given, so both can be trusted at
+          # once (the system roots, and this one extra CA).
+          params[:ca_file] = security.ca_file if security.ca_file
+          params[:ca_path] = security.ca_path if security.ca_path
+          context.set_params(params)
         else
           warn_once!
           context.set_params(verify_mode: ::OpenSSL::SSL::VERIFY_NONE)
         end
 
+        if security.client_cert && security.client_key
+          context.cert = coerce_cert(security.client_cert)
+          context.key = coerce_key(security.client_key)
+        end
+
         # Async::HTTP::Endpoint's own `alpn_protocols:` option only takes
         # effect on a context *it* builds internally — handing it a custom
         # ssl_context (as Butler always does, for verify_mode/CA control)
@@ -29,6 +44,30 @@ class Butler
         context
       end
 
+      # Accepts an already-built OpenSSL::X509::Certificate/PKey as-is (so a
+      # host that already parsed one, e.g. from a secrets manager, doesn't
+      # pay to re-parse it per connection), or a PEM string/file path.
+      def coerce_cert(value)
+        return value if value.is_a?(::OpenSSL::X509::Certificate)
+
+        ::OpenSSL::X509::Certificate.new(pem_source(value))
+      end
+
+      def coerce_key(value)
+        return value if value.is_a?(::OpenSSL::PKey::PKey)
+
+        ::OpenSSL::PKey.read(pem_source(value))
+      end
+
+      def pem_source(value)
+        ::File.exist?(value) ? ::File.read(value) : value
+      rescue ::ArgumentError, ::TypeError
+        # File.exist? raises ArgumentError on a string containing a NUL
+        # byte (a PEM blob never legitimately would) — treat it as literal
+        # PEM content rather than a path either way.
+        value
+      end
+
       def alpn_protocols_for(config)
         case config.http_version
         when :http1 then ["http/1.1"]

data/lib/butler/transport.rb

@@ -1,7 +1,10 @@
 require "async"
 require "async/http/client"
 require "async/http/endpoint"
+require "async/http/proxy"
 require "protocol/http/body/buffered"
+require "base64"
+require "uri"
 
 class Butler
   module Transport
@@ -48,23 +51,54 @@ class Butler
 
       def build_connection(uri, config)
         endpoint = build_endpoint(uri, config)
-        async_client = ::Async::HTTP::Client.new(endpoint, retries: 0)
-        Butler::Connection.new(origin_key: uri.origin_key, async_client: async_client, http_version: config.http_version)
+        proxy_client = nil
+
+        if config.proxy
+          endpoint, proxy_client = apply_proxy(endpoint, config.proxy)
+        end
+
+        # async-pool's own `limit:` (nil => unbounded, its default) — the
+        # single knob that actually bounds concurrent *connections* to this
+        # origin; ConnectionPool's own pool.max_connections is a different
+        # axis entirely (how many distinct origins stay warm at once).
+        async_client = ::Async::HTTP::Client.new(endpoint, retries: 0, limit: config.pool.max_connections_per_host)
+        Butler::Connection.new(origin_key: uri.origin_key, async_client: async_client, http_version: config.http_version, proxy_client: proxy_client)
       end
 
       def call(connection, request, config, timeout:)
         headers = request.headers.to_a
         body = build_request_body(request.body)
 
-        raw_response = Butler::Resilience::Timeout.enforce(timeout) do
-          connection.async_client.public_send(http_method(request.method), request.uri.request_target, headers, body)
+        # The full per-attempt budget has to cover reading the response
+        # body too, not just getting dispatch()'d back a raw_response —
+        # async-http returns as soon as the status line/headers are in
+        # hand, with the body available as a separate, lazily-read object.
+        # Wrapping only the dispatch call here previously let a server that
+        # sent headers promptly but then stalled (or drip-fed) the body
+        # hang for however long that actually took, completely ignoring
+        # deadline:/read_timeout/request_timeout — the one thing a request
+        # timeout exists to bound. The streaming case (request.stream?)
+        # is unaffected: build_response just wraps raw_response.body in a
+        # Butler::Stream there without reading anything yet, so this still
+        # only bounds getting the Stream object itself, exactly as
+        # documented — actual stream consumption stays the caller's own
+        # pacing, outside the attempt timeout, same as before.
+        Butler::Resilience::Timeout.enforce(timeout) do
+          raw_response = connection.async_client.public_send(http_method(request.method), request.uri.request_target, headers, body)
+          build_response(request, config, raw_response)
         end
-
-        build_response(request, config, raw_response)
       rescue ::Errno::ECONNREFUSED, ::Errno::ECONNRESET, ::Errno::EPIPE, ::EOFError, ::SocketError => e
         raise Butler::Errors::ConnectionError, "#{e.class}: #{e.message} requesting #{request.uri}"
       rescue ::OpenSSL::SSL::SSLError => e
         raise tls_error_class(e), "#{e.class}: #{e.message} requesting #{request.uri}"
+      rescue ::Async::HTTP::Proxy::ConnectFailure => e
+        # Raised by Async::HTTP::Proxy#connect when the proxy itself refuses
+        # the CONNECT tunnel (bad credentials, proxy down, target rejected) —
+        # only reachable once config.proxy is actually wired up (see
+        # apply_proxy above). Reclassified so `rescue Butler::Errors::Error`
+        # stays a safe top-level catch-all, same as every other transport
+        # failure here.
+        raise Butler::Errors::ConnectionError, "#{e.message} requesting #{request.uri} via proxy"
       end
 
       # OpenSSL's own verification-failure text is a stable, unified
@@ -88,6 +122,43 @@ class Butler
         ::Async::HTTP::Endpoint.parse(uri.origin_key, **options)
       end
 
+      # Wraps +endpoint+ so connecting to it tunnels through an HTTP(S)
+      # forward proxy via CONNECT (async-http's own Async::HTTP::Proxy,
+      # required above) instead of dialing the target directly — this is
+      # what actually makes config.proxy do anything; previously it was
+      # read into the connection-pool cache key but never passed to the
+      # transport at all. Works for both plain-HTTP and TLS targets: the
+      # CONNECT tunnel just carries bytes, and Endpoint itself layers TLS on
+      # top when the *target* URL is https, same as the non-proxied path.
+      #
+      # @returns [Array(Async::HTTP::Endpoint, Async::HTTP::Client)] the
+      #   proxied endpoint to hand to the real Async::HTTP::Client, and the
+      #   client holding the connection to the proxy server itself (kept
+      #   only so Butler::Connection#close can close it too).
+      def apply_proxy(endpoint, proxy)
+        proxy_uri = ::URI.parse(proxy)
+        proxy_endpoint = ::Async::HTTP::Endpoint.parse(
+          "#{proxy_uri.scheme}://#{proxy_uri.host}:#{proxy_uri.port}",
+        )
+        proxy_client = ::Async::HTTP::Client.new(proxy_endpoint, retries: 0)
+
+        [proxy_client.proxied_endpoint(endpoint, proxy_auth_headers(proxy_uri)), proxy_client]
+      end
+
+      # QuotaGuard-style proxy URLs (http://user:pass@host:port, the
+      # standard shape every "static outbound IP" add-on uses) carry
+      # credentials the proxy itself expects as Proxy-Authorization, not as
+      # part of the CONNECT target — URI#user/#password strip them out of
+      # proxy_uri.host/.port above, so they have to be re-attached here
+      # explicitly or they're silently dropped.
+      def proxy_auth_headers(proxy_uri)
+        return nil unless proxy_uri.user
+
+        user = ::URI.decode_www_form_component(proxy_uri.user)
+        password = ::URI.decode_www_form_component(proxy_uri.password.to_s)
+        [["proxy-authorization", "Basic #{::Base64.strict_encode64("#{user}:#{password}")}"]]
+      end
+
       def http_method(method)
         method.to_s.downcase.to_sym
       end
@@ -100,19 +171,33 @@ class Butler
 
       def build_response(request, config, raw_response)
         headers = Butler::Headers.from_a(raw_response.headers.to_a)
-        Butler::Security::Limits.check_headers!(headers, config)
+        close_on_error(raw_response) { Butler::Security::Limits.check_headers!(headers, config) }
 
         body =
           if request.stream?
             Butler::Stream.new { |&block| raw_response.body&.each(&block) }.on_close { raw_response.close }
           else
-            buffered = Butler::Security::Limits.read_body(raw_response.body, config)
+            buffered = close_on_error(raw_response) { Butler::Security::Limits.read_body(raw_response.body, config) }
             raw_response.close
             buffered
           end
 
         Butler::Response.new(uri: request.uri, status: raw_response.status, headers: headers, body: body)
       end
+
+      # Limits.check_headers!/read_body can both raise LimitExceeded before
+      # anything else has taken ownership of closing raw_response (the
+      # Stream case above only needs this via its own on_close — building it
+      # never raises). Without this, an oversized response leaks the
+      # underlying connection every time the limit trips: repeated hits
+      # (a misbehaving or hostile upstream) would eventually exhaust the
+      # connection pool/file descriptors instead of just failing the request.
+      def close_on_error(raw_response)
+        yield
+      rescue StandardError
+        raw_response.close
+        raise
+      end
     end
   end
 end

data/lib/butler/uri.rb

@@ -7,7 +7,20 @@ class Butler::URI
 
   def self.build(base_url, path = nil, params = nil)
     resolved = resolve(base_url, path)
-    new(::URI.parse(resolved)).with_params(params)
+
+    begin
+      parsed = ::URI.parse(resolved)
+    rescue ::URI::InvalidURIError => e
+      raise Butler::Errors::ConfigurationError, "invalid request URL #{resolved.inspect}: #{e.message}"
+    end
+
+    unless parsed.host
+      raise Butler::Errors::ConfigurationError,
+            "#{path.inspect} is not an absolute URL and no base_url is configured " \
+            "(set base_url: on Butler::Client.new/Butler.configure, or pass an absolute URL)"
+    end
+
+    new(parsed).with_params(params)
   end
 
   def self.resolve(base_url, path)

data/lib/butler/version.rb

@@ -1,3 +1,3 @@
 class Butler
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: butler-http
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Ram Laxman Yadav
@@ -93,6 +93,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '8.0'
+- !ruby/object:Gem::Dependency
+  name: elastic-transport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '8.5'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '8.5'
 description: |
   Butler is the modern Ruby HTTP client built for concurrent, resilient
   applications: Fiber-native concurrency, HTTP/1.1 + HTTP/2 with
@@ -116,6 +130,7 @@ files:
 - lib/butler/configuration.rb
 - lib/butler/connection.rb
 - lib/butler/connection_pool.rb
+- lib/butler/elasticsearch.rb
 - lib/butler/errors.rb
 - lib/butler/headers.rb
 - lib/butler/http.rb