httpx 1.7.5 → 1.7.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7e9530bb05303e7e15a05a84a26650fc2dd0ec0c35c11c909b07d8deb499f00f
-  data.tar.gz: 29043b781e3f12cf18ca80eeb6979833536b659dd13ce76d5312fe20f3b61342
+  metadata.gz: b34abe93b42862837d9131524d6e9663a93474f70c65ac2a10946eb62c9aeb14
+  data.tar.gz: '0853b845878005ace796106f79b735d9d75e205b7ab28c03a3786b53233d6a4b'
 SHA512:
-  metadata.gz: dea67bc736ed82b32c6c81257d66ca2dac2fae2cf41cb923fdca28c4c9741e5f57b0b56b3bfa23f6b0990f6faf26d14258b296ef771681d405564e02fafaceb5
-  data.tar.gz: 13dda593ac028e1d34b6ca0e22dd6a024705bf542da756a013f5da7f0abcccaab76860e80e5d8586ff95d4fe258542a295c004d33866a7986407c6cd4950f9e9
+  metadata.gz: 0be425ba5468990b31fb3fe04b97ee8c727cc1f16690decc2057314c3645e5baa21fae9b54585bc6e628410374dbfd9b8676df3aaed180b321a9b59cbbad2853
+  data.tar.gz: 3cbe9b0e2952c7d330bffe727f2b478f53152d431a306d48f5f41cb5033e9ba7fa2ddec9a319aed49f051c8e9e6e2d8e489ecccc83a6c622246ef6922c9bbecc

data/doc/release_notes/1_7_4.md

@@ -17,7 +17,7 @@ You can pass chain several tracers, and callbacks will be relayed to all of them
 HTTP.plugin(:tracing).with(tracer: telemetry_platform_tracer).with(tracer: telemetry2_platform_tracer)
 ```
 
-This was developed to be the foundation on top of which the datadag and OTel integrations will be built.
+This was developed to be the foundation on top of which the datadog and OTel integrations will be built.
 
 ## Improvements
 

data/doc/release_notes/1_7_6.md

@@ -0,0 +1,24 @@
+# 1.7.6
+
+## Improvements
+
+* `datadog` adapter: support setting custom `peer.service`.
+* stopped doing `Thread.pass` after checking connections back into the pool. The goal of this was to allow threads potentially waiting on a connection for the same origin to have the opportunity to take it before the same thread could check the same connection out, thereby preventing starvation on limited pool sizes; this however backfired for the common case of unbounded pool sizes (the default), and was specially bad when used for multiple concurrent requests on multiple origins, where avoidable lag was introduced on the whole processing every time a connection was checked back in. Thereby, the initial problem needs to be solved at the mutex implementation of CRuby.
+* connection: after keep alive timeout expired, try writing `PING` frame to the socket as soon as it's emitted (previously, the frame was being buffered and the socket would be probed for readiness before writing it, whereas in most cases this is avoidable).
+
+
+## Bugfixes
+
+* reuse the same resolver instance when resolving multiple concurrent requests in the same option (it was previously instantiating different ones based on option heuristics, such as when requests had different `:origin`).
+* on successful early name resolution (from cache or hosts file, for example), check the resolver back to the pool (the object was being dereferenced).
+* http1 parser: turn buffer back into a string on reset, instead of preserving its current type (in cases where it was a chunked transfer decoder and connection was kept-alive, it was keeping it around and using it to parse the next request, which broke).
+* http1 parser: enable pipelining based on the max concurrent requests, which the connection may have had pre-set to 1 based on prior knowledge of origin or other heuristics.
+* http1 parser: when disabling pipelining, mark incomplete inlined requests as idle before sending them back to the pending queue.
+* http1 and http2 parser: remove references to requests escalating an error; in scenarios of connection reuse (such as when using the `:persistent` plugin), these requests could be, on a transition to idle state, resent even in the absence of an external session reference, causing avoidable overhead or contention leading to timeouts or errors.
+* connection: rescue and ignore exceptions when calling `OpenSSL::SSL::SSLSocket#close` or `Socket#close`.
+* connection: when sending multiple requests into an open HTTP/2 connection which keep alive may have expired, only a single (instead of multiple, as before) `PING` frame will be sent (in the prior state, only the first `PING` was being acknowledged, which made the connection accumulate unacknowledged ping payloads).
+* connection: on handling errors, preventing potential race condition by copying pending requests into a local var before calling `parser.handle_error` (this may trigger a callback which calls `#reset`, which may call `#disconnect`, which would check the connection back into the pool and make it available for another thread, which could change the state of the ivar containing pending requests).
+* connection: skip resetting a connection when idle and having pending requests; this may happen in situation where parsers may have reset-and-back-to-idle the connection to resend failed requests as protocol (such as in the case of failed HTTP/1 pipelinining requests where incomplete requests are to be resent to the connection once pipelining is disabled).
+* connection: raise request timeout errors for the request the connection is currently in, instead of the connection where it was initially sent on and set timeouts on; this happens in situations where the request may be retried and sent to a different connection, which forces the request to keep a reference to the connection it's currently in (and discard it when it no longer needs it to prevent it from being GC'ed).
+* `:auth` plugin: when used alongside the `:retries` plugin for multiple concurrent requests, it was calling the dynamic token generator block once for each request; fixed to call the block only once per retry window, so retrying requests will reuse the same.
+* `:retries` plugin: retry request immediately if `:retry_after` is negative (while the option is validated for numbers, it can't control proc-based calculation).

data/lib/httpx/adapters/datadog.rb

@@ -19,6 +19,7 @@ module Datadog::Tracing
         Datadog::Tracing::Contrib::Ext::Metadata::TAG_BASE_SERVICE
       end
       TAG_PEER_HOSTNAME = Datadog::Tracing::Metadata::Ext::TAG_PEER_HOSTNAME
+      TAG_PEER_SERVICE = Datadog::Tracing::Metadata::Ext::TAG_PEER_SERVICE
 
       TAG_KIND = Datadog::Tracing::Metadata::Ext::TAG_KIND
       TAG_CLIENT = Datadog::Tracing::Metadata::Ext::SpanKind::TAG_CLIENT
@@ -108,7 +109,9 @@ module Datadog::Tracing
             span.set_tag(TAG_PEER_HOSTNAME, uri.host)
 
             # Tag as an external peer service
-            # span.set_tag(TAG_PEER_SERVICE, span.service)
+            if (peer_service = config[:peer_service])
+              span.set_tag(TAG_PEER_SERVICE, peer_service)
+            end
 
             if config[:distributed_tracing]
               propagate_trace_http(
@@ -210,6 +213,11 @@ module Datadog::Tracing
               o.env "DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE"
               o.default 1.0
             end
+
+            option :peer_service do |o|
+              o.type :string, nilable: true
+              o.env "DD_TRACE_HTTPX_PEER_SERVICE"
+            end
           else
             option :enabled do |o|
               o.default { env_to_bool("DD_TRACE_HTTPX_ENABLED", true) }
@@ -225,6 +233,11 @@ module Datadog::Tracing
               o.default { env_to_float(%w[DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE DD_HTTPX_ANALYTICS_SAMPLE_RATE], 1.0) }
               o.lazy
             end
+
+            option :peer_service do |o|
+              o.default { env_to_string("DD_TRACE_HTTPX_PEER_SERVICE", nil) }
+              o.lazy
+            end
           end
 
           if defined?(Datadog::Tracing::Contrib::SpanAttributeSchema)

data/lib/httpx/altsvc.rb

@@ -10,6 +10,8 @@ module HTTPX
 
       H2_ALTSVC_SCHEMES = %w[https h2].freeze
 
+      ALTSVC_IGNORE_IVARS = %i[@ssl].freeze
+
       def send(request)
         request.headers["alt-used"] = @origin.authority if @parser && !@write_buffer.full? && match_altsvcs?(request.uri)
 
@@ -35,11 +37,11 @@ module HTTPX
       end
 
       def match_altsvc_options?(uri, options)
-        return @options == options unless @options.ssl.all? do |k, v|
+        return @options.connection_options_match?(options) unless @options.ssl.all? do |k, v|
           v == (k == :hostname ? uri.host : options.ssl[k])
         end
 
-        @options.options_equals?(options, Options::REQUEST_BODY_IVARS + %i[@ssl])
+        @options.connection_options_match?(options, ALTSVC_IGNORE_IVARS)
       end
 
       def altsvc_match?(uri, other_uri)

data/lib/httpx/connection/http1.rb

@@ -28,7 +28,8 @@ module HTTPX
       @version = [1, 1]
       @pending = []
       @requests = []
-      @handshake_completed = false
+      @request = nil
+      @handshake_completed = @pipelining = false
     end
 
     def timeout
@@ -53,7 +54,9 @@ module HTTPX
     end
 
     def reset_requests
-      @pending.unshift(*@requests)
+      requests = @requests
+      requests.each { |r| r.transition(:idle) }
+      @pending.unshift(*requests)
       @requests.clear
     end
 
@@ -90,7 +93,7 @@ module HTTPX
       return if @requests.include?(request)
 
       @requests << request
-      @pipelining = true if @requests.size > 1
+      @pipelining = @max_concurrent_requests > 1 && @requests.size > 1
     end
 
     def consume
@@ -145,15 +148,17 @@ module HTTPX
 
       return unless request
 
-      log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
-      log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
-      response = request.response
+      begin
+        log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
+        log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
+        response = request.response
 
-      response << chunk
-    rescue StandardError => e
-      error_response = ErrorResponse.new(request, e)
-      request.response = error_response
-      dispatch
+        response << chunk
+      rescue StandardError => e
+        error_response = ErrorResponse.new(request, e)
+        request.response = error_response
+        dispatch(request)
+      end
     end
 
     def on_complete
@@ -162,12 +167,10 @@ module HTTPX
       return unless request
 
       log(level: 2) { "parsing complete" }
-      dispatch
+      dispatch(request)
     end
 
-    def dispatch
-      request = @request
-
+    def dispatch(request)
       if request.expects?
         @parser.reset!
         return handle(request)
@@ -217,12 +220,12 @@ module HTTPX
       if @pipelining
         catch(:called) { disable }
       else
-        @requests.each do |req|
+        while (req = @requests.shift)
           next if request && request == req
 
           emit(:error, req, ex)
         end
-        @pending.each do |req|
+        while (req = @pending.shift)
           next if request && request == req
 
           emit(:error, req, ex)

data/lib/httpx/connection/http2.rb

@@ -57,29 +57,37 @@ module HTTPX
 
         return if @buffer.empty?
 
+        # HTTP/2 GOAWAY frame buffered.
         return :w
       end
 
       unless @connection.state == :connected && @handshake_completed
+        # HTTP/2 in intermediate state or still completing initialization-
         return @buffer.empty? ? :r : :rw
       end
 
       unless @connection.send_buffer.empty?
+        # HTTP/2 connection is buffering data chunks and failing to emit DATA frames,
+        # most likely because the flow control window is exhausted.
         return :rw unless @buffer.empty?
 
         # waiting for WINDOW_UPDATE frames
         return :r
       end
 
+      # there are pending bufferable requests
       return :w if !@pending.empty? && can_buffer_more_requests?
 
+      # there are pending frames from the last run
       return :w unless @drains.empty?
 
       if @buffer.empty?
+        # skip if no more requests or pings to process
         return if @streams.empty? && @pings.empty?
 
         :r
       else
+        # buffered frames
         :w
       end
     end
@@ -138,7 +146,7 @@ module HTTPX
         settings_ex.set_backtrace(ex.backtrace)
         ex = settings_ex
       end
-      @streams.each_key do |req|
+      while (req, _ = @streams.shift)
         next if request && request == req
 
         emit(:error, req, ex)
@@ -399,10 +407,9 @@ module HTTPX
           ex = GoawayError.new(error)
           ex.set_backtrace(caller)
 
-          @pending.unshift(*@streams.keys)
+          handle_error(ex)
           teardown
 
-          handle_error(ex)
         end
       end
       return unless is_connection_closed && @streams.empty?

data/lib/httpx/connection.rb

@@ -62,6 +62,7 @@ module HTTPX
       @pending = []
       @inflight = 0
       @keep_alive_timeout = @options.timeout[:keep_alive_timeout]
+      @no_more_requests_counter = 0
 
       if @options.io
         # if there's an already open IO, get its
@@ -106,7 +107,7 @@ module HTTPX
         # origin came from an ORIGIN frame, we're going to verify the hostname with the
         # SSL certificate
         (@origins.size == 1 || @origin == uri.origin || (@io.is_a?(SSL) && @io.verify_hostname(uri.host))) &&
-        @options == options
+        @options.connection_options_match?(options)
     end
 
     def mergeable?(connection)
@@ -117,7 +118,7 @@ module HTTPX
       (
         (open? && @origin == connection.origin) ||
         !(@io.addresses & (connection.addresses || [])).empty?
-      ) && @options == connection.options
+      ) && @options.connection_options_match?(connection.options)
     end
 
     # coalesces +self+ into +connection+.
@@ -287,6 +288,10 @@ module HTTPX
     def reset
       return if @state == :closing || @state == :closed
 
+      # do not reset a connection which may have restarted back to :idle, such when the parser resets
+      # (example: HTTP/1 parser disabling pipelining)
+      return if @state == :idle && @pending.any?
+
       parser = @parser
 
       if parser && parser.respond_to?(:max_concurrent_requests)
@@ -313,8 +318,8 @@ module HTTPX
           log(level: 3) { "keep alive timeout expired, pinging connection..." }
           @pending << request
           transition(:active) if @state == :inactive
-          parser.ping
           request.ping!
+          ping
           return
         end
 
@@ -455,11 +460,11 @@ module HTTPX
           #
           # this condition takes into account:
           #
-          # * the number of inflight requests
           # * the number of pending requests
+          # * the number of inflight requests
           # * whether the write buffer has bytes (i.e. for close handshake)
           if @pending.empty? && @inflight.zero? && @write_buffer.empty?
-            log(level: 3) { "NO MORE REQUESTS..." } if @parser && @parser.pending.any?
+            no_more_requests_loop_check if @parser && @parser.pending.any?
 
             # terminate if an altsvc connection has been established
             terminate if @altsvc_connection
@@ -509,7 +514,7 @@ module HTTPX
 
             # exit #consume altogether if all outstanding requests have been dealt with
             if @pending.empty? && @inflight.zero? && @write_buffer.empty? # rubocop:disable Style/Next
-              log(level: 3) { "NO MORE REQUESTS..." } if @parser && @parser.pending.any?
+              no_more_requests_loop_check if @parser && @parser.pending.any?
 
               # terminate if an altsvc connection has been established
               terminate if @altsvc_connection
@@ -635,6 +640,7 @@ module HTTPX
           build_altsvc_connection(alt_origin, origin, alt_params)
         end
         @response_received_at = Utils.now
+        @no_more_requests_counter = 0
         @inflight -= 1
         response.finish!
         request.emit(:response, response)
@@ -643,7 +649,7 @@ module HTTPX
         build_altsvc_connection(alt_origin, origin, alt_params)
       end
 
-      parser.on(:pong, &method(:send_pending))
+      parser.on(:pong, &method(:pong))
 
       parser.on(:promise) do |request, stream|
         request.emit(:promise, parser, stream)
@@ -755,16 +761,6 @@ module HTTPX
         return if @inflight.positive? || @parser.waiting_for_ping?
       when :closing
         return unless connecting? || @state == :open
-
-        unless @write_buffer.empty?
-          # preset state before handshake, as error callbacks
-          # may take it back here.
-          @state = nextstate
-          # handshakes, try sending
-          consume
-          @write_buffer.clear
-          return
-        end
       when :closed
         return unless @state == :closing
         return unless @write_buffer.empty?
@@ -790,6 +786,12 @@ module HTTPX
       case nextstate
       when :inactive
         disconnect
+      when :closing
+        return if @write_buffer.empty?
+
+        # try flushing termination handshakes
+        consume
+        @write_buffer.clear
       when :closed
         # TODO: should this raise an error instead?
         return unless @pending.empty?
@@ -813,16 +815,18 @@ module HTTPX
     end
 
     def close_sibling
-      return unless @sibling
+      sibling = @sibling
 
-      if @sibling.io_connected?
+      return unless sibling
+
+      if sibling.io_connected?
         reset
         # TODO: transition connection to closed
       end
 
-      unless @sibling.state == :closed
-        merge(@sibling) unless @main_sibling
-        @sibling.force_reset(true)
+      unless sibling.state == :closed
+        merge(sibling) unless @main_sibling
+        sibling.force_reset(true)
       end
 
       @sibling = nil
@@ -902,28 +906,61 @@ module HTTPX
       end
     end
 
+    def ping
+      return if parser.waiting_for_ping?
+
+      parser.ping
+      call
+    end
+
+    def pong
+      @response_received_at = Utils.now
+      @no_more_requests_counter = 0
+      send_pending
+    end
+
+    def no_more_requests_loop_check
+      log(level: 3) { "NO MORE REQUESTS..." }
+      @no_more_requests_counter += 1
+
+      return if @no_more_requests_counter < 50
+
+      raise Error, "connection corrupted, aborted after looping for a while, " \
+                   "please report this https://gitlab.com/os85/httpx/-/work_items " \
+                   "along with debug logs"
+    end
+
     # recover internal state and emit all relevant error responses when +error+ was raised.
     # this takes an optiona +request+ which may have already been handled and can be opted out
     # in the state recovery process.
     def handle_error(error, request = nil)
-      parser.handle_error(error, request) if @parser && @parser.respond_to?(:handle_error)
-      while (req = @pending.shift)
-        next if request && req == request
+      if request
+        @inflight -= 1
+        response = ErrorResponse.new(request, error)
+        request.response = response
+        request.emit(:response, response)
+      end
 
-        response = ErrorResponse.new(req, error)
-        req.response = response
-        req.emit(:response, response)
+      pending = @pending
+      if (parser = @parser) && parser.respond_to?(:handle_error)
+        # parser.handle_error may disconnect the connection
+        pending = @pending.dup
+        @pending = []
+
+        parser.handle_error(error, request)
       end
 
-      return unless request
+      while (req = pending.shift)
+        next if request && req == request
 
-      @inflight -= 1
-      response = ErrorResponse.new(request, error)
-      request.response = response
-      request.emit(:response, response)
+        resp = ErrorResponse.new(req, error)
+        req.response = resp
+        req.emit(:response, resp)
+      end
     end
 
     def set_request_timeouts(request)
+      request.connection = self
       set_request_write_timeout(request)
       set_request_read_timeout(request)
       set_request_request_timeout(request)
@@ -959,29 +996,29 @@ module HTTPX
       end
     end
 
-    def write_timeout_callback(request, write_timeout)
+    def write_timeout_callback(request, timeout)
       return if request.state == :done
 
       @write_buffer.clear
-      error = WriteTimeoutError.new(request, nil, write_timeout)
+      error = WriteTimeoutError.new(request, nil, timeout)
 
-      on_error(error, request)
+      request.handle_error(error)
     end
 
-    def read_timeout_callback(request, read_timeout, error_type = ReadTimeoutError)
+    def read_timeout_callback(request, timeout, error_type = ReadTimeoutError)
       response = request.response
 
       return if response && response.finished?
 
       @write_buffer.clear
-      error = error_type.new(request, request.response, read_timeout)
+      error = error_type.new(request, response, timeout)
 
-      on_error(error, request)
+      request.handle_error(error)
     end
 
     def set_request_timeout(label, request, timeout, start_event, finish_events, &callback)
       request.set_timeout_callback(start_event) do
-        unless @current_selector
+        unless (selector = @current_selector)
           raise Error, "request has been resend to an out-of-session connection, and this " \
                        "should never happen!!! Please report this error! " \
                        "(state:#{@state}, " \
@@ -992,7 +1029,7 @@ module HTTPX
                        "coalesced?:#{coalesced?})"
         end
 
-        timer = @current_selector.after(timeout, callback)
+        timer = selector.after(timeout, callback)
         request.active_timeouts << label
 
         Array(finish_events).each do |event|

data/lib/httpx/io/ssl.rb

@@ -51,6 +51,7 @@ module HTTPX
     end
 
     def protocol
+      # @type ivar @io: OpenSSL::SSL::SSLSocket
       @io.alpn_protocol || super
     rescue StandardError
       super
@@ -60,6 +61,7 @@ module HTTPX
       # in jruby, alpn_protocol may return ""
       # https://github.com/jruby/jruby-openssl/issues/287
       def protocol
+        # @type ivar @io: OpenSSL::SSL::SSLSocket
         proto = @io.alpn_protocol
 
         return super if proto.nil? || proto.empty?
@@ -76,9 +78,10 @@ module HTTPX
 
     def verify_hostname(host)
       return false if @ctx.verify_mode == OpenSSL::SSL::VERIFY_NONE
-      return false if !@io.respond_to?(:peer_cert) || @io.peer_cert.nil?
+      # @type ivar @io: OpenSSL::SSL::SSLSocket
+      return false if !@io.respond_to?(:peer_cert) || (peer_cert = @io.peer_cert).nil?
 
-      OpenSSL::SSL.verify_certificate_identity(@io.peer_cert, host)
+      OpenSSL::SSL.verify_certificate_identity(peer_cert, host)
     end
 
     def connected?
@@ -86,7 +89,9 @@ module HTTPX
     end
 
     def ssl_session_expired?
-      @ssl_session.nil? || Process.clock_gettime(Process::CLOCK_REALTIME) >= (@ssl_session.time.to_f + @ssl_session.timeout)
+      ssl_session = @ssl_session
+
+      ssl_session.nil? || Process.clock_gettime(Process::CLOCK_REALTIME) >= (ssl_session.time.to_f + ssl_session.timeout)
     end
 
     def connect
@@ -97,6 +102,8 @@ module HTTPX
         return unless @state == :connected
       end
 
+      # @type ivar @io: OpenSSL::SSL::SSLSocket
+
       unless @io.is_a?(OpenSSL::SSL::SSLSocket)
         if (hostname_is_ip = (@ip == @sni_hostname)) && @ctx.verify_hostname
           # IPv6 address would be "[::1]", must turn to "0000:0000:0000:0000:0000:0000:0000:0001" for cert SAN check
@@ -105,16 +112,19 @@ module HTTPX
           @ctx.verify_hostname = false
         end
 
-        @io = OpenSSL::SSL::SSLSocket.new(@io, @ctx)
+        ssl = OpenSSL::SSL::SSLSocket.new(@io, @ctx)
 
-        @io.hostname = @sni_hostname unless hostname_is_ip
-        @io.session = @ssl_session unless ssl_session_expired?
-        @io.sync_close = true
+        ssl.hostname = @sni_hostname unless hostname_is_ip
+        ssl.session = @ssl_session unless ssl_session_expired?
+        ssl.sync_close = true
+
+        @io = ssl
       end
       try_ssl_connect
     end
 
     def try_ssl_connect
+      # @type ivar @io: OpenSSL::SSL::SSLSocket
       ret = @io.connect_nonblock(exception: false)
       log(level: 3, color: :cyan) { "TLS CONNECT: #{ret}..." }
       case ret
@@ -147,7 +157,9 @@ module HTTPX
     def log_transition_state(nextstate)
       return super unless nextstate == :negotiated
 
-      server_cert = @io.peer_cert
+      # @type ivar @io: OpenSSL::SSL::SSLSocket
+
+      server_cert = @io.peer_cert #: OpenSSL::X509::Certificate
 
       "#{super}\n\n" \
         "SSL connection using #{@io.ssl_version} / #{Array(@io.cipher).first}\n" \

data/lib/httpx/io/tcp.rb

@@ -23,18 +23,21 @@ module HTTPX
       @fallback_protocol = @options.fallback_protocol
       @port = origin.port
       @interests = :w
-      if @options.io
-        @io = case @options.io
-              when Hash
-                @options.io[origin.authority]
-              else
-                @options.io
-        end
-        raise Error, "Given IO objects do not match the request authority" unless @io
-
-        _, _, _, ip = @io.addr
-        @ip = Resolver::Entry.new(ip)
-        @addresses << @ip
+      if (io = @options.io)
+        io =
+          case io
+          when Hash
+            io[origin.authority]
+          else
+            io
+          end
+        raise Error, "Given IO objects do not match the request authority" unless io
+
+        # @type var io: TCPSocket | OpenSSL::SSL::SSLSocket
+
+        _, _, _, ip = io.addr
+        @io = io
+        @addresses << (@ip = Resolver::Entry.new(ip))
         @keep_open = true
         @state = :connected
       else
@@ -183,6 +186,9 @@ module HTTPX
 
       begin
         @io.close
+      rescue StandardError => e
+        log { "error closing socket" }
+        log { e.full_message(highlight: false) }
       ensure
         transition(:closed)
       end

data/lib/httpx/io/unix.rb

@@ -14,16 +14,20 @@ module HTTPX
       @state = :idle
       @options = options
       @fallback_protocol = @options.fallback_protocol
-      if @options.io
-        @io = case @options.io
-              when Hash
-                @options.io[origin.authority]
-              else
-                @options.io
-        end
-        raise Error, "Given IO objects do not match the request authority" unless @io
+      if (io = @options.io)
+        io =
+          case io
+          when Hash
+            io[origin.authority]
+          else
+            io
+          end
+        raise Error, "Given IO objects do not match the request authority" unless io
+
+        # @type var io: UNIXSocket
 
-        @path = @io.path
+        _, @path = io.addr
+        @io = io
         @keep_open = true
         @state = :connected
       elsif path

data/lib/httpx/options.rb

@@ -202,16 +202,19 @@ module HTTPX
 
     REQUEST_BODY_IVARS = %i[@headers].freeze
 
-    def ==(other)
-      super || options_equals?(other)
-    end
+    # checks whether +other+ matches the same connection-level options
+    def connection_options_match?(other, ignore_ivars = nil)
+      return true if self == other
 
-    # checks whether +other+ is equal by comparing the session options
-    def options_equals?(other, ignore_ivars = REQUEST_BODY_IVARS)
       # headers and other request options do not play a role, as they are
       # relevant only for the request.
-      ivars = instance_variables - ignore_ivars
-      other_ivars = other.instance_variables - ignore_ivars
+      ivars = instance_variables
+      ivars.reject! { |iv| REQUEST_BODY_IVARS.include?(iv) }
+      ivars.reject! { |iv| ignore_ivars.include?(iv) } if ignore_ivars
+
+      other_ivars = other.instance_variables
+      other_ivars.reject! { |iv| REQUEST_BODY_IVARS.include?(iv) }
+      other_ivars.reject! { |iv| ignore_ivars.include?(iv) } if ignore_ivars
 
       return false if ivars.size != other_ivars.size
 
@@ -222,6 +225,19 @@ module HTTPX
       end
     end
 
+    RESOLVER_IVARS = %i[
+      @resolver_class @resolver_cache @resolver_options
+      @resolver_native_class @resolver_system_class @resolver_https_class
+    ].freeze
+
+    # checks whether +other+ matches the same resolver-level options
+    def resolver_options_match?(other)
+      self == other ||
+        RESOLVER_IVARS.all? do |ivar|
+          instance_variable_get(ivar) == other.instance_variable_get(ivar)
+        end
+    end
+
     # returns a HTTPX::Options instance resulting of the merging of +other+ with self.
     # it may return self if +other+ is self or equal to self.
     def merge(other)

data/lib/httpx/parser/http1.rb

@@ -14,6 +14,8 @@ module HTTPX
         @state = :idle
         @buffer = "".b
         @headers = {}
+        @content_length = nil
+        @_has_trailers = @upgrade = false
       end
 
       def <<(chunk)
@@ -25,7 +27,8 @@ module HTTPX
         @state = :idle
         @headers = {}
         @content_length = nil
-        @_has_trailers = nil
+        @_has_trailers = @upgrade = false
+        @buffer = @buffer.to_s
         @buffer.clear
       end
 
@@ -34,7 +37,7 @@ module HTTPX
       end
 
       def upgrade_data
-        @buffer
+        @buffer.to_s
       end
 
       private
@@ -55,6 +58,7 @@ module HTTPX
       end
 
       def parse_headline
+        #: @type ivar @buffer: String
         idx = @buffer.index("\n")
         return unless idx
 
@@ -75,6 +79,8 @@ module HTTPX
         headers = @headers
         buffer = @buffer
 
+        #: @type var buffer: String
+
         while (idx = buffer.index("\n"))
           # @type var line: String
           line = buffer.byteslice(0..idx)
@@ -118,17 +124,20 @@ module HTTPX
 
       def parse_data
         if @buffer.respond_to?(:each)
+          # @type ivar @buffer: Transcoder::Chunker::Decoder
           @buffer.each do |chunk|
             @observer.on_data(chunk)
           end
         elsif @content_length
-          # @type var data: String
+          # @type ivar @buffer: String
           data = @buffer.byteslice(0, @content_length)
+          # @type var data: String
           @buffer = @buffer.byteslice(@content_length..-1) || "".b
           @content_length -= data.bytesize
           @observer.on_data(data)
           data.clear
         else
+          # @type ivar @buffer: String
           @observer.on_data(@buffer)
           @buffer.clear
         end
@@ -152,7 +161,7 @@ module HTTPX
             tr_encoding.split(/ *, */).each do |encoding|
               case encoding
               when "chunked"
-                @buffer = Transcoder::Chunker::Decoder.new(@buffer, @_has_trailers)
+                @buffer = Transcoder::Chunker::Decoder.new(@buffer.to_s, @_has_trailers)
               end
             end
           end
@@ -165,6 +174,7 @@ module HTTPX
         if @content_length
           @content_length <= 0
         elsif @buffer.respond_to?(:finished?)
+          # @type ivar @buffer: Transcoder::Chunker::Decoder
           @buffer.finished?
         else
           false

data/lib/httpx/plugins/auth.rb

@@ -44,6 +44,7 @@ module HTTPX
           super
 
           @auth_header_value = nil
+          @auth_header_value_mtx = Thread::Mutex.new
           @skip_auth_header_value = false
         end
 
@@ -63,7 +64,9 @@ module HTTPX
         end
 
         def reset_auth_header_value!
-          @auth_header_value = nil
+          @auth_header_value_mtx.synchronize do
+            @auth_header_value = nil
+          end
         end
 
         private
@@ -71,9 +74,11 @@ module HTTPX
         def send_request(request, *)
           return super if @skip_auth_header_value || request.authorized?
 
-          @auth_header_value ||= generate_auth_token
+          auth_header_value = @auth_header_value_mtx.synchronize do
+            @auth_header_value ||= generate_auth_token
+          end
 
-          request.authorize(@auth_header_value) if @auth_header_value
+          request.authorize(auth_header_value) if auth_header_value
 
           super
         end
@@ -92,9 +97,11 @@ module HTTPX
       end
 
       module RequestMethods
+        attr_reader :auth_token_value
+
         def initialize(*)
           super
-          @auth_token_value = nil
+          @auth_token_value = @auth_header_value = nil
         end
 
         def authorized?
@@ -102,19 +109,20 @@ module HTTPX
         end
 
         def unauthorize!
-          return unless (auth_value = @auth_token_value)
+          return unless (auth_value = @auth_header_value)
 
           @headers.get("authorization").delete(auth_value)
 
-          @auth_token_value = nil
+          @auth_token_value = @auth_header_value = nil
         end
 
         def authorize(auth_value)
+          @auth_header_value = auth_value
           if (auth_type = @options.auth_header_type)
-            auth_value = "#{auth_type} #{auth_value}"
+            @auth_header_value = "#{auth_type} #{@auth_header_value}"
           end
 
-          @headers.add("authorization", auth_value)
+          @headers.add("authorization", @auth_header_value)
 
           @auth_token_value = auth_value
         end
@@ -138,8 +146,14 @@ module HTTPX
             return unless auth_error?(response, request.options) ||
                           (@options.generate_auth_value_on_retry && @options.generate_auth_value_on_retry.call(response))
 
+            # regenerate token before retry, but only if it's the first request from batch failing.
+            # otherwise, it means that the first request already passed here, so this request should
+            # use whatever was generated for it.
+            @auth_header_value_mtx.synchronize do
+              @auth_header_value = generate_auth_token if request.auth_token_value == @auth_header_value
+            end
+
             request.unauthorize!
-            @auth_header_value = generate_auth_token
           end
 
           def auth_error?(response, options)

data/lib/httpx/plugins/retries.rb

@@ -148,7 +148,7 @@ module HTTPX
             log { "failed to get response, #{request.retries} tries to go..." }
             prepare_to_retry(request, response)
 
-            if (retry_after = when_to_retry(request, response, options))
+            if (retry_after = when_to_retry(request, response, options)) && retry_after.positive?
               # apply jitter
               if (jitter = request.options.retry_jitter)
                 retry_after = jitter.call(retry_after)

data/lib/httpx/pool.rb

@@ -122,12 +122,6 @@ module HTTPX
 
         @max_connections_cond.signal
         @origin_conds[connection.origin.to_s].signal
-
-        # Observed situations where a session handling multiple requests in a loop
-        # across multiple threads checks the same connection in and out, while another
-        # thread which is waiting on the same connection never gets the chance to pick
-        # it up, because ruby's thread scheduler never switched on to it in the process.
-        Thread.pass
       end
     end
 
@@ -153,16 +147,20 @@ module HTTPX
         resolvers = @resolvers[resolver_type]
 
         idx = resolvers.find_index do |res|
-          res.options == options
+          res.options.resolver_options_match?(options)
         end
         resolvers.delete_at(idx) if idx
       end || checkout_new_resolver(resolver_type, options)
     end
 
     def checkin_resolver(resolver)
-      resolver_class = resolver.class
+      if resolver.is_a?(Resolver::Multi)
+        resolver_class = resolver.resolvers.first.class
+      else
+        resolver_class = resolver.class
 
-      resolver = resolver.multi
+        resolver = resolver.multi
+      end
 
       # a multi requires all sub-resolvers being closed in order to be
       # correctly checked back in.

data/lib/httpx/request.rb

@@ -42,6 +42,9 @@ module HTTPX
     # The IP address from the peer server.
     attr_accessor :peer_address
 
+    # the connection the request is currently being sent to (none if before or after transaction)
+    attr_writer :connection
+
     attr_writer :persistent
 
     attr_reader :active_timeouts
@@ -91,7 +94,7 @@ module HTTPX
       raise UnsupportedSchemeError, "#{@uri}: #{@uri.scheme}: unsupported URI scheme" unless ALLOWED_URI_SCHEMES.include?(@uri.scheme)
 
       @state = :idle
-      @response = @drainer = @peer_address = @informational_status = nil
+      @connection = @response = @drainer = @peer_address = @informational_status = nil
       @ping = false
       @persistent = @options.persistent
       @active_timeouts = []
@@ -274,8 +277,7 @@ module HTTPX
       when :idle
         @body.rewind
         @ping = false
-        @response = nil
-        @drainer = nil
+        @response = @drainer = nil
         @active_timeouts.clear
       when :headers
         return unless @state == :idle
@@ -321,6 +323,16 @@ module HTTPX
         callbacks(event).delete(clb)
       end
     end
+
+    def handle_error(error)
+      if (connection = @connection)
+        connection.on_error(error, self)
+      else
+        response = ErrorResponse.new(self, error)
+        self.response = response
+        emit(:response, response)
+      end
+    end
   end
 end
 

data/lib/httpx/resolver/native.rb

@@ -66,7 +66,7 @@ module HTTPX
     end
 
     def closed?
-      @state == :closed
+      @state == :idle || @state == :closed
     end
 
     def to_io

data/lib/httpx/response.rb

@@ -112,6 +112,7 @@ module HTTPX
     def finish!
       @finished = true
       @headers.freeze
+      @request.connection = nil
     end
 
     # returns whether the response contains body payload.
@@ -282,6 +283,7 @@ module HTTPX
       @error = error
       @options = request.options
       log_exception(@error)
+      finish!
     end
 
     # returns the exception full message.
@@ -299,7 +301,9 @@ module HTTPX
       true
     end
 
-    def finish!; end
+    def finish!
+      @request.connection = nil
+    end
 
     # raises the wrapped exception.
     def raise_for_status

data/lib/httpx/selector.rb

@@ -81,7 +81,8 @@ module HTTPX
 
     def find_resolver(options)
       res = @selectables.find do |c|
-        c.is_a?(Resolver::Resolver) && options == c.options
+        c.is_a?(Resolver::Resolver) &&
+          options.resolver_options_match?(c.options)
       end
 
       res.multi if res
@@ -148,16 +149,18 @@ module HTTPX
 
         next(is_closed) if is_closed
 
-        io.log(level: 2) do
-          "[#{io.state}] registering in selector##{object_id} for select (#{interests})#{" for #{interval} seconds" unless interval.nil?}"
-        end
+        if interests
+          io.log(level: 2) do
+            "[#{io.state}] registering in selector##{object_id} for select (#{interests})#{" for #{interval} seconds" unless interval.nil?}"
+          end
 
-        if READABLE.include?(interests)
-          r = r.nil? ? io : (Array(r) << io)
-        end
+          if READABLE.include?(interests)
+            r = r.nil? ? io : (Array(r) << io)
+          end
 
-        if WRITABLE.include?(interests)
-          w = w.nil? ? io : (Array(w) << io)
+          if WRITABLE.include?(interests)
+            w = w.nil? ? io : (Array(w) << io)
+          end
         end
 
         is_closed

data/lib/httpx/session.rb

@@ -373,7 +373,11 @@ module HTTPX
       resolver = find_resolver_for(connection, selector)
 
       pin(connection, selector)
-      early_resolve(resolver, connection) || resolver.lazy_resolve(connection)
+      if early_resolve(resolver, connection)
+        @pool.checkin_resolver(resolver)
+      else
+        resolver.lazy_resolve(connection)
+      end
     end
 
     def early_resolve(resolver, connection)

data/lib/httpx/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module HTTPX
-  VERSION = "1.7.5"
+  VERSION = "1.7.6"
 end

data/sig/altsvc.rbs

@@ -3,6 +3,8 @@ module HTTPX
     module ConnectionMixin
       H2_ALTSVC_SCHEMES: Array[String]
 
+      ALTSVC_IGNORE_IVARS: Array[Symbol]
+
       def send: (Request request) -> void
 
       def match?: (http_uri uri, Options options) -> bool

data/sig/connection/http1.rbs

@@ -52,7 +52,7 @@ module HTTPX
 
     def on_complete: () -> void
 
-    def dispatch: () -> void
+    def dispatch: (Request request) -> void
 
     def ping: () -> void
 

data/sig/connection.rbs

@@ -50,6 +50,7 @@ module HTTPX
     @altsvc_connection: instance?
     @sibling: instance?
     @main_sibling: bool
+    @no_more_requests_counter: Integer
 
 
     def addresses: () -> Array[Resolver::Entry]?
@@ -156,6 +157,12 @@ module HTTPX
 
     def build_socket: (?Array[Resolver::Entry]? addrs) -> (TCP | SSL | UNIX)
 
+    def ping: () -> void
+
+    def pong: () -> void
+
+    def no_more_requests_loop_check: () -> void
+
     def handle_error: (StandardError error, ?Request? request) -> void
 
     def force_purge: () -> void
@@ -172,9 +179,9 @@ module HTTPX
 
     def set_request_request_timeout: (Request request) -> void
 
-    def write_timeout_callback: (Request request, Numeric write_timeout) -> void
+    def write_timeout_callback: (Request request, Numeric timeout) -> void
 
-    def read_timeout_callback: (Request request, Numeric read_timeout, ?singleton(RequestTimeoutError) error_type) -> void
+    def read_timeout_callback: (Request request, Numeric timeout, ?singleton(RequestTimeoutError) error_type) -> void
 
     def set_request_timeout: (Symbol label, Request request, Numeric timeout, Symbol start_event, Symbol | Array[Symbol] finish_events) { () -> void } -> void
 

data/sig/io/ssl.rbs

@@ -8,6 +8,7 @@ module HTTPX
 
     @ctx: OpenSSL::SSL::SSLContext
     @verify_hostname: bool
+    @sni_hostname: String
 
     attr_writer ssl_session: OpenSSL::SSL::Session?
 

data/sig/io/tcp.rbs

@@ -46,7 +46,7 @@ module HTTPX
 
     public
 
-    def read: (Integer size, ?(Buffer | String) buffer) -> (0 | nil | untyped)
+    def read: (Integer size, Buffer | String buffer) -> (0 | nil | untyped)
 
     def write: (Buffer buffer) -> Integer?
 
@@ -67,6 +67,6 @@ module HTTPX
 
     def do_transition: (Symbol nextstate) -> void
 
-    def log_transition_state: (Symbol nextstate) -> void
+    def log_transition_state: (Symbol nextstate) -> String
   end
 end

data/sig/options.rbs

@@ -17,6 +17,7 @@ module HTTPX
 
     DEFAULT_OPTIONS: Hash[Symbol, untyped]
     REQUEST_BODY_IVARS: Array[Symbol]
+    RESOLVER_IVARS: Array[Symbol]
     RESOLVER_TYPES: Array[Symbol]
     USER_AGENT: String
 
@@ -106,8 +107,10 @@ module HTTPX
 
     attr_reader ssl: Hash[Symbol, untyped]
 
-    # io
-    type io_option = _ToIO | Hash[String, _ToIO]
+    type external_io = TCPSocket | OpenSSL::SSL::SSLSocket | UNIXSocket
+
+    type io_option = external_io | Hash[String, external_io]
+
     attr_reader io: io_option?
 
     # fallback_protocol
@@ -142,7 +145,9 @@ module HTTPX
 
     def ==: (Options other) -> bool
 
-    def options_equals?: (Options other, ?Array[Symbol] ignore_ivars) -> bool
+    def connection_options_match?: (Options other, ?Array[Symbol] ignore_ivars) -> bool
+
+    def resolver_options_match?: (Options other) -> bool
 
     def merge: (Object & _ToHash[Symbol, untyped] other) -> (instance | self)
 

data/sig/parser/http1.rbs

@@ -23,7 +23,7 @@ module HTTPX
 
       @observer: _HTTP1Events
       @state: Symbol
-      @buffer: String
+      @buffer: String | Transcoder::Chunker::Decoder
       @content_type: String?
       @content_length: Integer
       @_has_trailers: bool

data/sig/plugins/auth.rbs

@@ -10,7 +10,8 @@ module HTTPX
       end
 
       module InstanceMethods
-        @auth_header_value: String
+        @auth_header_value: String?
+        @auth_header_value_mtx: Thread::Mutex
         @skip_auth_header_value: bool
 
         def authorization: (?string token, ?auth_header_type: string) ?{ (Request) -> string } -> instance
@@ -29,7 +30,9 @@ module HTTPX
       end
 
       module RequestMethods
-        @auth_token_value: String?
+        attr_reader auth_token_value: String?
+
+        @auth_header_value: String?
 
         def authorized?: () -> bool
 

data/sig/plugins/retries.rbs

@@ -47,7 +47,7 @@ module HTTPX
 
         def prepare_to_retry: (Request & RequestMethods request, retriesResponse response) -> void
 
-        def when_to_retry: (Request & RequestMethods request, retriesResponse response, retriesOptions options) -> void
+        def when_to_retry: (Request & RequestMethods request, retriesResponse response, retriesOptions options) -> Numeric?
       end
 
       module RequestMethods

data/sig/pool.rbs

@@ -34,7 +34,7 @@ module HTTPX
 
     def checkout_resolver: (Options options) -> resolver_manager
 
-    def checkin_resolver: (Resolver::Resolver resolver) -> void
+    def checkin_resolver: (Resolver::Multi | Resolver::Resolver resolver) -> void
 
     private
 

data/sig/request.rbs

@@ -18,6 +18,7 @@ module HTTPX
     attr_reader drain_error: StandardError?
     attr_reader active_timeouts: Array[Symbol]
 
+    attr_writer connection: Connection?
     attr_accessor peer_address: (String | IPAddr)?
 
     attr_writer persistent: bool
@@ -81,6 +82,8 @@ module HTTPX
 
     def set_timeout_callback: (Symbol event) { (*untyped) -> void } -> void
 
+    def handle_error: (StandardError error) -> void
+
     private
 
     def initialize_body: (Options options) -> Transcoder::_Encoder?

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: httpx
 version: !ruby/object:Gem::Version
-  version: 1.7.5
+  version: 1.7.6
 platform: ruby
 authors:
 - Tiago Cardoso
@@ -165,6 +165,7 @@ extra_rdoc_files:
 - doc/release_notes/1_7_3.md
 - doc/release_notes/1_7_4.md
 - doc/release_notes/1_7_5.md
+- doc/release_notes/1_7_6.md
 files:
 - LICENSE.txt
 - README.md
@@ -302,6 +303,7 @@ files:
 - doc/release_notes/1_7_3.md
 - doc/release_notes/1_7_4.md
 - doc/release_notes/1_7_5.md
+- doc/release_notes/1_7_6.md
 - lib/httpx.rb
 - lib/httpx/adapters/datadog.rb
 - lib/httpx/adapters/faraday.rb