httpx 1.7.5 → 1.7.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7e9530bb05303e7e15a05a84a26650fc2dd0ec0c35c11c909b07d8deb499f00f
-  data.tar.gz: 29043b781e3f12cf18ca80eeb6979833536b659dd13ce76d5312fe20f3b61342
+  metadata.gz: 64a496153f6b67985861f18f1ab86793d356bfb1f49045346e55c4df8a991881
+  data.tar.gz: 449bcbe6d7acf74e744f296dbfdf2530fe411b66bfecfba5552e5f6185465ea0
 SHA512:
-  metadata.gz: dea67bc736ed82b32c6c81257d66ca2dac2fae2cf41cb923fdca28c4c9741e5f57b0b56b3bfa23f6b0990f6faf26d14258b296ef771681d405564e02fafaceb5
-  data.tar.gz: 13dda593ac028e1d34b6ca0e22dd6a024705bf542da756a013f5da7f0abcccaab76860e80e5d8586ff95d4fe258542a295c004d33866a7986407c6cd4950f9e9
+  metadata.gz: '02781b0d9ce574816cb07722aa5f22383c75872b2d782ba9bfc651b9b94b5ffb03860f01cfebdd2de5d38d28e40f7388595b751d549f7ec791e5bf4a9fd3b9e9'
+  data.tar.gz: 7f5de973f936e7fa6265918d7d8facf2602367c85cbc90c3ad09b92b98c01f34f9f9513a0faff0498f945d12cb41938b232bf74b96a45eb03ab680fd721ed43f

data/doc/release_notes/1_7_4.md

@@ -17,7 +17,7 @@ You can pass chain several tracers, and callbacks will be relayed to all of them
 HTTP.plugin(:tracing).with(tracer: telemetry_platform_tracer).with(tracer: telemetry2_platform_tracer)
 ```
 
-This was developed to be the foundation on top of which the datadag and OTel integrations will be built.
+This was developed to be the foundation on top of which the datadog and OTel integrations will be built.
 
 ## Improvements
 

data/doc/release_notes/1_7_6.md

@@ -0,0 +1,24 @@
+# 1.7.6
+
+## Improvements
+
+* `datadog` adapter: support setting custom `peer.service`.
+* stopped doing `Thread.pass` after checking connections back into the pool. The goal of this was to allow threads potentially waiting on a connection for the same origin to have the opportunity to take it before the same thread could check the same connection out, thereby preventing starvation on limited pool sizes; this however backfired for the common case of unbounded pool sizes (the default), and was specially bad when used for multiple concurrent requests on multiple origins, where avoidable lag was introduced on the whole processing every time a connection was checked back in. Thereby, the initial problem needs to be solved at the mutex implementation of CRuby.
+* connection: after keep alive timeout expired, try writing `PING` frame to the socket as soon as it's emitted (previously, the frame was being buffered and the socket would be probed for readiness before writing it, whereas in most cases this is avoidable).
+
+
+## Bugfixes
+
+* reuse the same resolver instance when resolving multiple concurrent requests in the same option (it was previously instantiating different ones based on option heuristics, such as when requests had different `:origin`).
+* on successful early name resolution (from cache or hosts file, for example), check the resolver back to the pool (the object was being dereferenced).
+* http1 parser: turn buffer back into a string on reset, instead of preserving its current type (in cases where it was a chunked transfer decoder and connection was kept-alive, it was keeping it around and using it to parse the next request, which broke).
+* http1 parser: enable pipelining based on the max concurrent requests, which the connection may have had pre-set to 1 based on prior knowledge of origin or other heuristics.
+* http1 parser: when disabling pipelining, mark incomplete inlined requests as idle before sending them back to the pending queue.
+* http1 and http2 parser: remove references to requests escalating an error; in scenarios of connection reuse (such as when using the `:persistent` plugin), these requests could be, on a transition to idle state, resent even in the absence of an external session reference, causing avoidable overhead or contention leading to timeouts or errors.
+* connection: rescue and ignore exceptions when calling `OpenSSL::SSL::SSLSocket#close` or `Socket#close`.
+* connection: when sending multiple requests into an open HTTP/2 connection which keep alive may have expired, only a single (instead of multiple, as before) `PING` frame will be sent (in the prior state, only the first `PING` was being acknowledged, which made the connection accumulate unacknowledged ping payloads).
+* connection: on handling errors, preventing potential race condition by copying pending requests into a local var before calling `parser.handle_error` (this may trigger a callback which calls `#reset`, which may call `#disconnect`, which would check the connection back into the pool and make it available for another thread, which could change the state of the ivar containing pending requests).
+* connection: skip resetting a connection when idle and having pending requests; this may happen in situation where parsers may have reset-and-back-to-idle the connection to resend failed requests as protocol (such as in the case of failed HTTP/1 pipelinining requests where incomplete requests are to be resent to the connection once pipelining is disabled).
+* connection: raise request timeout errors for the request the connection is currently in, instead of the connection where it was initially sent on and set timeouts on; this happens in situations where the request may be retried and sent to a different connection, which forces the request to keep a reference to the connection it's currently in (and discard it when it no longer needs it to prevent it from being GC'ed).
+* `:auth` plugin: when used alongside the `:retries` plugin for multiple concurrent requests, it was calling the dynamic token generator block once for each request; fixed to call the block only once per retry window, so retrying requests will reuse the same.
+* `:retries` plugin: retry request immediately if `:retry_after` is negative (while the option is validated for numbers, it can't control proc-based calculation).

data/doc/release_notes/1_7_7.md

@@ -0,0 +1,17 @@
+# 1.7.7
+
+## Improvements
+
+* ssl cert info will also be redacted when `:debug_redact` option is enabled.
+* fix performance regression of #receive_requests recent loop change (which also improved performance of previous baseline).
+* header/data logs indexed by request (instead of parser).
+
+## Bugfixes
+
+* proxy: fixed issue when basic auth to proxy failed with 407 and it'd raise NoMethodError when retrying (marked as unretriable now).
+* selector: prevent removal of wrong selectable when previous closed via `#on_close` during selectables traversal.
+* http1: only reset requests which weren't completed (prevent issuing errors twice).
+* http1: forego clearing buffer on reset (avoid deleting potential next request bytes).
+* response_cache: reset cached response age on successful cache revalidation, as per RFC.
+* rate_limiter: do not apply jitter when delay comes from `retry-after` header.
+* digest: digest header parsing issues results in an error response (instead of an exception raised).

data/lib/httpx/adapters/datadog.rb

@@ -19,6 +19,7 @@ module Datadog::Tracing
         Datadog::Tracing::Contrib::Ext::Metadata::TAG_BASE_SERVICE
       end
       TAG_PEER_HOSTNAME = Datadog::Tracing::Metadata::Ext::TAG_PEER_HOSTNAME
+      TAG_PEER_SERVICE = Datadog::Tracing::Metadata::Ext::TAG_PEER_SERVICE
 
       TAG_KIND = Datadog::Tracing::Metadata::Ext::TAG_KIND
       TAG_CLIENT = Datadog::Tracing::Metadata::Ext::SpanKind::TAG_CLIENT
@@ -108,7 +109,9 @@ module Datadog::Tracing
             span.set_tag(TAG_PEER_HOSTNAME, uri.host)
 
             # Tag as an external peer service
-            # span.set_tag(TAG_PEER_SERVICE, span.service)
+            if (peer_service = config[:peer_service])
+              span.set_tag(TAG_PEER_SERVICE, peer_service)
+            end
 
             if config[:distributed_tracing]
               propagate_trace_http(
@@ -210,6 +213,11 @@ module Datadog::Tracing
               o.env "DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE"
               o.default 1.0
             end
+
+            option :peer_service do |o|
+              o.type :string, nilable: true
+              o.env "DD_TRACE_HTTPX_PEER_SERVICE"
+            end
           else
             option :enabled do |o|
               o.default { env_to_bool("DD_TRACE_HTTPX_ENABLED", true) }
@@ -225,6 +233,11 @@ module Datadog::Tracing
               o.default { env_to_float(%w[DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE DD_HTTPX_ANALYTICS_SAMPLE_RATE], 1.0) }
               o.lazy
             end
+
+            option :peer_service do |o|
+              o.default { env_to_string("DD_TRACE_HTTPX_PEER_SERVICE", nil) }
+              o.lazy
+            end
           end
 
           if defined?(Datadog::Tracing::Contrib::SpanAttributeSchema)

data/lib/httpx/adapters/faraday.rb

@@ -182,15 +182,6 @@ module Faraday
               @on_response
             end
           end
-
-          def on_complete(&blk)
-            if blk
-              @on_complete = blk
-              self
-            else
-              @on_complete
-            end
-          end
         end
 
         include RequestMixin
@@ -224,7 +215,6 @@ module Faraday
             Array(responses).each_with_index do |response, index|
               handler = @handlers[index]
               handler.on_response.call(response)
-              handler.on_complete.call(handler.env) if handler.on_complete
             end
           end
         end

data/lib/httpx/adapters/webmock.rb

@@ -132,7 +132,7 @@ module WebMock
             request.transition(:done)
             response.finish!
             request.response = response
-            request.emit(:response, response)
+            request.emit_response(response)
             request_signature.headers = request.headers.to_h
 
             response << mock_response.body.dup unless response.is_a?(HTTPX::ErrorResponse)

data/lib/httpx/altsvc.rb

@@ -10,6 +10,8 @@ module HTTPX
 
       H2_ALTSVC_SCHEMES = %w[https h2].freeze
 
+      ALTSVC_IGNORE_IVARS = %i[@ssl].freeze
+
       def send(request)
         request.headers["alt-used"] = @origin.authority if @parser && !@write_buffer.full? && match_altsvcs?(request.uri)
 
@@ -35,11 +37,11 @@ module HTTPX
       end
 
       def match_altsvc_options?(uri, options)
-        return @options == options unless @options.ssl.all? do |k, v|
+        return @options.connection_options_match?(options) unless @options.ssl.all? do |k, v|
           v == (k == :hostname ? uri.host : options.ssl[k])
         end
 
-        @options.options_equals?(options, Options::REQUEST_BODY_IVARS + %i[@ssl])
+        @options.connection_options_match?(options, ALTSVC_IGNORE_IVARS)
       end
 
       def altsvc_match?(uri, other_uri)

data/lib/httpx/connection/http1.rb

@@ -28,7 +28,8 @@ module HTTPX
       @version = [1, 1]
       @pending = []
       @requests = []
-      @handshake_completed = false
+      @request = nil
+      @handshake_completed = @pipelining = false
     end
 
     def timeout
@@ -53,7 +54,12 @@ module HTTPX
     end
 
     def reset_requests
-      @pending.unshift(*@requests)
+      @requests.reverse_each do |request|
+        next if request.response
+
+        request.transition(:idle)
+        @pending.unshift(request)
+      end
       @requests.clear
     end
 
@@ -90,7 +96,7 @@ module HTTPX
       return if @requests.include?(request)
 
       @requests << request
-      @pipelining = true if @requests.size > 1
+      @pipelining = @max_concurrent_requests > 1 && @requests.size > 1
     end
 
     def consume
@@ -113,30 +119,32 @@ module HTTPX
     end
 
     def on_headers(h)
-      @request = @requests.first
+      request = @request = @requests.first
 
-      return if @request.response
+      return if request.response
 
-      log(level: 2) { "headers received" }
-      headers = @request.options.headers_class.new(h)
-      response = @request.options.response_class.new(@request,
-                                                     @parser.status_code,
-                                                     @parser.http_version.join("."),
-                                                     headers)
-      log(color: :yellow) { "-> HEADLINE: #{response.status} HTTP/#{@parser.http_version.join(".")}" }
-      log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v)}" }.join("\n") }
+      request.log(level: 2) { "headers received" }
+      headers = request.options.headers_class.new(h)
+      response = request.options.response_class.new(request,
+                                                    @parser.status_code,
+                                                    @parser.http_version.join("."),
+                                                    headers)
+      request.log(color: :yellow) { "-> HEADLINE: #{response.status} HTTP/#{@parser.http_version.join(".")}" }
+      request.log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v)}" }.join("\n") }
 
-      @request.response = response
+      request.response = response
       on_complete if response.finished?
     end
 
     def on_trailers(h)
-      return unless @request
+      request = @request
 
-      response = @request.response
-      log(level: 2) { "trailer headers received" }
+      return unless request
 
-      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v.join(", "))}" }.join("\n") }
+      response = request.response
+
+      request.log(level: 2) { "trailer headers received" }
+      request.log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v.join(", "))}" }.join("\n") }
       response.merge_headers(h)
     end
 
@@ -145,15 +153,18 @@ module HTTPX
 
       return unless request
 
-      log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
-      log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
-      response = request.response
+      begin
+        request.log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
+        request.log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
+
+        response = request.response
 
-      response << chunk
-    rescue StandardError => e
-      error_response = ErrorResponse.new(request, e)
-      request.response = error_response
-      dispatch
+        response << chunk
+      rescue StandardError => e
+        error_response = ErrorResponse.new(request, e)
+        request.response = error_response
+        dispatch(request)
+      end
     end
 
     def on_complete
@@ -161,13 +172,11 @@ module HTTPX
 
       return unless request
 
-      log(level: 2) { "parsing complete" }
-      dispatch
+      request.log(level: 2) { "parsing complete" }
+      dispatch(request)
     end
 
-    def dispatch
-      request = @request
-
+    def dispatch(request)
       if request.expects?
         @parser.reset!
         return handle(request)
@@ -217,12 +226,12 @@ module HTTPX
       if @pipelining
         catch(:called) { disable }
       else
-        @requests.each do |req|
+        while (req = @requests.shift)
           next if request && request == req
 
           emit(:error, req, ex)
         end
-        @pending.each do |req|
+        while (req = @pending.shift)
           next if request && request == req
 
           emit(:error, req, ex)
@@ -290,7 +299,7 @@ module HTTPX
       return if @max_concurrent_requests == 1
 
       @requests.each do |r|
-        r.transition(:idle)
+        r.transition(:idle) if r.response.nil?
 
         # when we disable pipelining, we still want to try keep-alive.
         # only when keep-alive with one request fails, do we fallback to
@@ -359,10 +368,10 @@ module HTTPX
     def join_headers(request)
       headline = join_headline(request)
       @buffer << headline << CRLF
-      log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
+      request.log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
       extra_headers = set_protocol_headers(request)
-      join_headers2(request.headers.each(extra_headers))
-      log { "<- " }
+      join_headers2(request, request.headers.each(extra_headers))
+      request.log { "<- " }
       @buffer << CRLF
     end
 
@@ -370,8 +379,8 @@ module HTTPX
       return if request.body.empty?
 
       while (chunk = request.drain_body)
-        log(color: :green) { "<- DATA: #{chunk.bytesize} bytes..." }
-        log(level: 2, color: :green) { "<- #{log_redact_body(chunk.inspect)}" }
+        request.log(color: :green) { "<- DATA: #{chunk.bytesize} bytes..." }
+        request.log(level: 2, color: :green) { "<- #{log_redact_body(chunk.inspect)}" }
         @buffer << chunk
         throw(:buffer_full, request) if @buffer.full?
       end
@@ -384,15 +393,15 @@ module HTTPX
     def join_trailers(request)
       return unless request.trailers? && request.callbacks_for?(:trailers)
 
-      join_headers2(request.trailers)
-      log { "<- " }
+      join_headers2(request, request.trailers)
+      request.log { "<- " }
       @buffer << CRLF
     end
 
-    def join_headers2(headers)
+    def join_headers2(request, headers)
       headers.each do |field, value|
         field = capitalized(field)
-        log(color: :yellow) { "<- HEADER: #{[field, log_redact_headers(value)].join(": ")}" }
+        request.log(color: :yellow) { "<- HEADER: #{[field, log_redact_headers(value)].join(": ")}" }
         @buffer << "#{field}: #{value}#{CRLF}"
       end
     end

data/lib/httpx/connection/http2.rb

@@ -57,29 +57,37 @@ module HTTPX
 
         return if @buffer.empty?
 
+        # HTTP/2 GOAWAY frame buffered.
         return :w
       end
 
       unless @connection.state == :connected && @handshake_completed
+        # HTTP/2 in intermediate state or still completing initialization-
         return @buffer.empty? ? :r : :rw
       end
 
       unless @connection.send_buffer.empty?
+        # HTTP/2 connection is buffering data chunks and failing to emit DATA frames,
+        # most likely because the flow control window is exhausted.
         return :rw unless @buffer.empty?
 
         # waiting for WINDOW_UPDATE frames
         return :r
       end
 
+      # there are pending bufferable requests
       return :w if !@pending.empty? && can_buffer_more_requests?
 
+      # there are pending frames from the last run
       return :w unless @drains.empty?
 
       if @buffer.empty?
+        # skip if no more requests or pings to process
         return if @streams.empty? && @pings.empty?
 
         :r
       else
+        # buffered frames
         :w
       end
     end
@@ -138,7 +146,7 @@ module HTTPX
         settings_ex.set_backtrace(ex.backtrace)
         ex = settings_ex
       end
-      @streams.each_key do |req|
+      while (req, _ = @streams.shift)
         next if request && request == req
 
         emit(:error, req, ex)
@@ -234,11 +242,11 @@ module HTTPX
       extra_headers = set_protocol_headers(request)
 
       if request.headers.key?("host")
-        log { "forbidden \"host\" header found (#{log_redact_headers(request.headers["host"])}), will use it as authority..." }
+        request.log { "forbidden \"host\" header found (#{log_redact_headers(request.headers["host"])}), will use it as authority..." }
         extra_headers[":authority"] = request.headers["host"]
       end
 
-      log(level: 1, color: :yellow) do
+      request.log(level: 1, color: :yellow) do
         "\n#{request.headers.merge(extra_headers).each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")}"
       end
       stream.headers(request.headers.each(extra_headers), end_stream: request.body.empty?)
@@ -250,7 +258,7 @@ module HTTPX
         return
       end
 
-      log(level: 1, color: :yellow) do
+      request.log(level: 1, color: :yellow) do
         request.trailers.each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")
       end
       stream.headers(request.trailers.each, end_stream: true)
@@ -278,8 +286,8 @@ module HTTPX
     end
 
     def send_chunk(request, stream, chunk, next_chunk)
-      log(level: 1, color: :green) { "#{stream.id}: -> DATA: #{chunk.bytesize} bytes..." }
-      log(level: 2, color: :green) { "#{stream.id}: -> #{log_redact_body(chunk.inspect)}" }
+      request.log(level: 1, color: :green) { "#{stream.id}: -> DATA: #{chunk.bytesize} bytes..." }
+      request.log(level: 2, color: :green) { "#{stream.id}: -> #{log_redact_body(chunk.inspect)}" }
       stream.data(chunk, end_stream: end_stream?(request, next_chunk))
     end
 
@@ -295,11 +303,11 @@ module HTTPX
       response = request.response
 
       if response.is_a?(Response) && response.version == "2.0"
-        on_stream_trailers(stream, response, h)
+        on_stream_trailers(stream, request, response, h)
         return
       end
 
-      log(color: :yellow) do
+      request.log(color: :yellow) do
         h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{k == ":status" ? v : log_redact_headers(v)}" }.join("\n")
       end
       _, status = h.shift
@@ -311,16 +319,16 @@ module HTTPX
       handle(request, stream) if request.expects?
     end
 
-    def on_stream_trailers(stream, response, h)
-      log(color: :yellow) do
+    def on_stream_trailers(stream, request, response, h)
+      request.log(color: :yellow) do
         h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")
       end
       response.merge_headers(h)
     end
 
     def on_stream_data(stream, request, data)
-      log(level: 1, color: :green) { "#{stream.id}: <- DATA: #{data.bytesize} bytes..." }
-      log(level: 2, color: :green) { "#{stream.id}: <- #{log_redact_body(data.inspect)}" }
+      request.log(level: 1, color: :green) { "#{stream.id}: <- DATA: #{data.bytesize} bytes..." }
+      request.log(level: 2, color: :green) { "#{stream.id}: <- #{log_redact_body(data.inspect)}" }
       request.response << data
     end
 
@@ -329,14 +337,14 @@ module HTTPX
       stream.close
     end
 
-    def on_stream_half_close(stream, _request)
+    def on_stream_half_close(stream, request)
       unless stream.send_buffer.empty?
         stream.send_buffer.clear
         stream.data("", end_stream: true)
       end
 
       # TODO: omit log line if response already here
-      log(level: 2) { "#{stream.id}: waiting for response..." }
+      request.log(level: 2) { "#{stream.id}: waiting for response..." }
     end
 
     def on_stream_close(stream, request, error)
@@ -399,10 +407,9 @@ module HTTPX
           ex = GoawayError.new(error)
           ex.set_backtrace(caller)
 
-          @pending.unshift(*@streams.keys)
+          handle_error(ex)
           teardown
 
-          handle_error(ex)
         end
       end
       return unless is_connection_closed && @streams.empty?