httpx 1.6.1 → 1.6.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 947d1ad4fd269f30707292be54687f7be2054147fb3fa471c76395e92c55820e
-  data.tar.gz: d6e321ace4211e8bf8215eff799b5c98aad627990db5b9b2cc091fe5b1b95df7
+  metadata.gz: f49b29ea3703f6f40abe3cd82d455235b0a0b50a694bd8fa55839ac471d32bbb
+  data.tar.gz: ce933bb3c35d9434f810d4fc83acb244a1ad0db187123d5f01dcbbbd83622a47
 SHA512:
-  metadata.gz: bd87db725b6da121b79f266a8a309816f5afea4ec1890a14720c16a667ea32632fdbdd9d74249b337672315fd2ac16c1488d8aa13f4e031b33751e1625f2c9a5
-  data.tar.gz: 00027b822dff3985519aba56eb7b3bd28d0d81eacbdeb9d49f6e778023d60ae3eaf1a2469cf11287da12c0dde2a23b1557da33e77310ea6733e03e4f73b822e2
+  metadata.gz: 258cb32129840347a1a37633fb6273133187c98636b63ffcc0b8064f39e3b5642d693ee5da5d7776529625fbe4b68b5193b405c2e0658b2c9b1193b49e316bdc
+  data.tar.gz: 387397ab1954b6abf6a8a54d8764fb25e786fb1bd293401e8afd57f823464f67210527fee0b9e94b34cc3d2931143a5d94598fb6241f5728df77a8f1e045a5bf

data/doc/release_notes/1_6_2.md

@@ -0,0 +1,11 @@
+# 1.6.2
+
+## Bugfixes
+
+* revert of behaviour introduced in 1.6.1 around handling of `:ip_families` for name resolution.
+  * when no option is passed, do not assume no IPv6 connectivity if no available non-local IP is found, as the local network may still be reachable under `[::]`.
+* bail out connection if the tcp connection was established but the ssl session failed.
+  * when alpn negotiation succeeded, this could still initialize the HTTP/2 connection and put bytes in the buffer, which would cause a busy loop trying to write to a non-open socket.
+* datadog: fix initialization of spans in non-connection related errors
+  * past code was relying on the error being around DNS, but other errors could pop; the fix was moving the init time setup early to the session, when a request is first passed ot the associated connection.
+  * it can also fail earlier, so provide a workaround for that as well.

data/doc/release_notes/1_6_3.md

@@ -0,0 +1,47 @@
+# 1.6.3
+
+## Features
+
+* allow redacting only headers, or only the body, when using `debug_redact: :headers` or `debug_redact: :body` respectively.
+
+## Improvements
+
+* `system` resolver now works in a non-blocking manner, initiating the dns query in a separate thread and waiting on the pipe after that (it was blocking the main thread during resolution before).
+* reduce allocation to a sinfle shared option object when headers are passed as a session-level option, like `HTTPX.with(headers: geaders).get(...)`
+* privilege using `String#replace` in buffer operations (instead of "clean-then-append").
+* using `Array#unshift` instead of `Array#concat` in order to ensure that request ordering is respected in the face of an in-between error which requires reconnect-and-resend.
+* replaced more internal callback indirection with plain method calls.
+
+
+## Bugfixes
+
+* https: prevent modification of the ssl context object when performing a reconnection.
+* compression: do not return early if the decompression buffer yields an empty string (more frequent under jruby 10).
+* response cache: take query params into account when caching or retrieving cached responses.
+* response cache: do not decompress cached responses on body consumption (the response bodies are cached in plaintext).
+* native resolver: pick next timeout associated with the hostname being resolved (and not the hostnames in the queue).
+* pool: assume that, even when signalled that a connection is available, context may be switched to a session which also checks the same connection out, before it's able to pick it up; in such a case, start from the beginning, until the pool timeout expires.
+* session: forego bookkeeping when a connection is coalesced (instead, allow it to be dropped).
+* digest_auth: make sure that an array is sent back if the probe response fails.
+* alt-svc: when alt-svc handshake happens with more in-flight requests, defer termination to when these requests are made.
+* http2: fix use of unexisting var `ex` when processing the connection closed callback.
+* connection: fix potential session dereferencing, which allowed connections to be used across sessions, therefore bypassing needed synchronization and leading to the `undefined method 'after' for nil:NilClass` error.
+* selector: close only selected connections (instead of all selectable connections) when an error occurs during IO readiness wait calls.
+* resolvers: correctly propagate abrupt termination errors to the connection objects waiting for the answer.
+* resolvers: when errors happenm force-close unresolved connections (and ensure they're both pinned to the corresponding session before the error happens, and are unpinned after error is propagated).
+* resolvers: ensure resolvers transition to "closed" state, on all cases, when any error happens.
+* resolvers: ensure that the next hostname is resolved when a timeout happens on the current one.
+* native resolver: fixed duplication of the hostname to resolve in the list of candidates.
+* https resolver: use a `system` resolver to resolve the DoH server hostname (instead of rerouting it to itself).
+* https resolver: skip loop error reporting when error happens outside of it.
+* https resolver: close connection on resolve errors, which prevents it from being around in the pool after termination; also deactivate it after successful use.
+* multi resolver: do not check resolvers back into the pool if it's a multi resolver and the peer is still resolving (and do the check outside of the critical area).
+* sentry adapter: removed usage of deprecated method which has been removed in sentry-ruby 6.0.0.
+* selector: when coalescing connections, pin the current session before merging connections, to prevent it from registering in a selector being used in a different thread, and inadvertedly allowing it to be used across threads.
+* session: fix: always pin connection before early-or-lazy resolution (fixes connection pool accounting under connection coalescing).
+
+## Chores
+
+* logging emits a timestamp as well (to monitor timeouts).
+* `:stream_bidi` plugin: extends HTTP2 module by using plugin extensions.
+* connection: remove session/selector references when closing a connection (prevents leaking them beyond the usage scope).

data/lib/httpx/adapters/datadog.rb

@@ -180,7 +180,7 @@ module Datadog::Tracing
         end
 
         module RequestMethods
-          attr_reader :init_time
+          attr_accessor :init_time
 
           # intercepts request initialization to inject the tracing logic.
           def initialize(*)
@@ -193,26 +193,30 @@ module Datadog::Tracing
             RequestTracer.call(self)
           end
 
-          def response=(response)
-            if response.is_a?(::HTTPX::ErrorResponse) && response.error.respond_to?(:connection)
-              # handles the case when the +error+ happened during name resolution, which means
-              # that the tracing start point hasn't been triggered yet; in such cases, the approximate
-              # initial resolving time is collected from the connection, and used as span start time,
-              # and the tracing object in inserted before the on response callback is called.
-              @init_time = response.error.connection.init_time
-            end
+          def response=(*)
+            # init_time should be set when it's send to a connection.
+            # However, there are situations where connection initialization fails.
+            # Example is the :ssrf_filter plugin, which raises an error on
+            # initialize if the host is an IP which matches against the known set.
+            # in such cases, we'll just set here right here.
+            @init_time ||= ::Datadog::Core::Utils::Time.now.utc
+
             super
           end
         end
 
         module ConnectionMethods
-          attr_reader :init_time
-
           def initialize(*)
             super
 
             @init_time = ::Datadog::Core::Utils::Time.now.utc
           end
+
+          def send(request)
+            request.init_time ||= @init_time
+
+            super
+          end
         end
       end
 

data/lib/httpx/adapters/sentry.rb

@@ -32,7 +32,7 @@ module HTTPX::Plugins
 
         return unless config.propagate_traces && config.trace_propagation_targets.any? { |target| url.match?(target) }
 
-        trace = ::Sentry.get_current_client.generate_sentry_trace(sentry_span)
+        trace = sentry_span.to_sentry_trace
         request.headers[::Sentry::SENTRY_TRACE_HEADER_NAME] = trace if trace
       end
 

data/lib/httpx/connection/http1.rb

@@ -44,12 +44,12 @@ module HTTPX
       @max_requests = @options.max_requests || MAX_REQUESTS
       @parser.reset!
       @handshake_completed = false
-      @pending.concat(@requests) unless @requests.empty?
+      @pending.unshift(*@requests)
     end
 
     def close
       reset
-      emit(:close, true)
+      emit(:close)
     end
 
     def exhausted?
@@ -114,7 +114,7 @@ module HTTPX
                                                      @parser.http_version.join("."),
                                                      headers)
       log(color: :yellow) { "-> HEADLINE: #{response.status} HTTP/#{@parser.http_version.join(".")}" }
-      log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{log_redact(v)}" }.join("\n") }
+      log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v)}" }.join("\n") }
 
       @request.response = response
       on_complete if response.finished?
@@ -126,7 +126,7 @@ module HTTPX
       response = @request.response
       log(level: 2) { "trailer headers received" }
 
-      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{log_redact(v.join(", "))}" }.join("\n") }
+      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v.join(", "))}" }.join("\n") }
       response.merge_headers(h)
     end
 
@@ -136,7 +136,7 @@ module HTTPX
       return unless request
 
       log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
-      log(level: 2, color: :green) { "-> #{log_redact(chunk.inspect)}" }
+      log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
       response = request.response
 
       response << chunk
@@ -182,7 +182,7 @@ module HTTPX
       end
 
       if exhausted?
-        @pending.concat(@requests)
+        @pending.unshift(*@requests)
         @requests.clear
 
         emit(:exhausted)
@@ -236,7 +236,7 @@ module HTTPX
       when /keep-alive/i
         if @handshake_completed
           if @max_requests.zero?
-            @pending.concat(@requests)
+            @pending.unshift(*@requests)
             @requests.clear
             emit(:exhausted)
           end
@@ -360,7 +360,7 @@ module HTTPX
 
       while (chunk = request.drain_body)
         log(color: :green) { "<- DATA: #{chunk.bytesize} bytes..." }
-        log(level: 2, color: :green) { "<- #{log_redact(chunk.inspect)}" }
+        log(level: 2, color: :green) { "<- #{log_redact_body(chunk.inspect)}" }
         @buffer << chunk
         throw(:buffer_full, request) if @buffer.full?
       end
@@ -381,7 +381,7 @@ module HTTPX
     def join_headers2(headers)
       headers.each do |field, value|
         field = capitalized(field)
-        log(color: :yellow) { "<- HEADER: #{[field, log_redact(value)].join(": ")}" }
+        log(color: :yellow) { "<- HEADER: #{[field, log_redact_headers(value)].join(": ")}" }
         @buffer << "#{field}: #{value}#{CRLF}"
       end
     end

data/lib/httpx/connection/http2.rb

@@ -89,7 +89,7 @@ module HTTPX
         @connection.goaway
         emit(:timeout, @options.timeout[:close_handshake_timeout])
       end
-      emit(:close, true)
+      emit(:close)
     end
 
     def empty?
@@ -234,12 +234,12 @@ module HTTPX
       extra_headers = set_protocol_headers(request)
 
       if request.headers.key?("host")
-        log { "forbidden \"host\" header found (#{log_redact(request.headers["host"])}), will use it as authority..." }
+        log { "forbidden \"host\" header found (#{log_redact_headers(request.headers["host"])}), will use it as authority..." }
         extra_headers[":authority"] = request.headers["host"]
       end
 
       log(level: 1, color: :yellow) do
-        "\n#{request.headers.merge(extra_headers).each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{log_redact(v)}" }.join("\n")}"
+        "\n#{request.headers.merge(extra_headers).each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")}"
       end
       stream.headers(request.headers.each(extra_headers), end_stream: request.body.empty?)
     end
@@ -251,7 +251,7 @@ module HTTPX
       end
 
       log(level: 1, color: :yellow) do
-        request.trailers.each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{log_redact(v)}" }.join("\n")
+        request.trailers.each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")
       end
       stream.headers(request.trailers.each, end_stream: true)
     end
@@ -279,7 +279,7 @@ module HTTPX
 
     def send_chunk(request, stream, chunk, next_chunk)
       log(level: 1, color: :green) { "#{stream.id}: -> DATA: #{chunk.bytesize} bytes..." }
-      log(level: 2, color: :green) { "#{stream.id}: -> #{log_redact(chunk.inspect)}" }
+      log(level: 2, color: :green) { "#{stream.id}: -> #{log_redact_body(chunk.inspect)}" }
       stream.data(chunk, end_stream: end_stream?(request, next_chunk))
     end
 
@@ -300,7 +300,7 @@ module HTTPX
       end
 
       log(color: :yellow) do
-        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{log_redact(v)}" }.join("\n")
+        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")
       end
       _, status = h.shift
       headers = request.options.headers_class.new(h)
@@ -313,14 +313,14 @@ module HTTPX
 
     def on_stream_trailers(stream, response, h)
       log(color: :yellow) do
-        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{log_redact(v)}" }.join("\n")
+        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")
       end
       response.merge_headers(h)
     end
 
     def on_stream_data(stream, request, data)
       log(level: 1, color: :green) { "#{stream.id}: <- DATA: #{data.bytesize} bytes..." }
-      log(level: 2, color: :green) { "#{stream.id}: <- #{log_redact(data.inspect)}" }
+      log(level: 2, color: :green) { "#{stream.id}: <- #{log_redact_body(data.inspect)}" }
       request.response << data
     end
 
@@ -387,18 +387,17 @@ module HTTPX
           end
         else
           ex = GoawayError.new(error)
+          ex.set_backtrace(caller)
+
           @pending.unshift(*@streams.keys)
           teardown
-        end
 
-        if ex
-          ex.set_backtrace(caller)
           handle_error(ex)
         end
       end
       return unless is_connection_closed && @streams.empty?
 
-      emit(:close, is_connection_closed)
+      emit(:close) if is_connection_closed
     end
 
     def on_frame_sent(frame)
@@ -409,7 +408,7 @@ module HTTPX
           when :data
             frame.merge(payload: frame[:payload].bytesize)
           when :headers, :ping
-            frame.merge(payload: log_redact(frame[:payload]))
+            frame.merge(payload: log_redact_headers(frame[:payload]))
           else
             frame
           end
@@ -425,7 +424,7 @@ module HTTPX
           when :data
             frame.merge(payload: frame[:payload].bytesize)
           when :headers, :ping
-            frame.merge(payload: log_redact(frame[:payload]))
+            frame.merge(payload: log_redact_headers(frame[:payload]))
           else
             frame
           end
@@ -435,7 +434,7 @@ module HTTPX
 
     def on_altsvc(origin, frame)
       log(level: 2) { "#{frame[:stream]}: altsvc frame was received" }
-      log(level: 2) { "#{frame[:stream]}: #{log_redact(frame.inspect)}" }
+      log(level: 2) { "#{frame[:stream]}: #{log_redact_headers(frame.inspect)}" }
       alt_origin = URI.parse("#{frame[:proto]}://#{frame[:host]}:#{frame[:port]}")
       params = { "ma" => frame[:max_age] }
       emit(:altsvc, origin, alt_origin, origin, params)