httpx 1.4.0 → 1.4.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7edc783221b5d919a1c788bf360f0cc2d43ee1556eb893c6b5d377d1bd3b4241
-  data.tar.gz: 14dc4593ae5c46acb33a4f242a0058e75eddec70bd22fdf75c05496240de3abc
+  metadata.gz: 4aa5b1cfd43d05f50fc7246219d6a90cb5591e11c8bfeee1f1cf62ffa91993fc
+  data.tar.gz: 58f96325ac13b8caddb708464a75ee6dafcfe7b8284f0825803ac3d0c4d8ccc1
 SHA512:
-  metadata.gz: 181bc9e6155708c2d1cdaee3dc7c5c15cd5e2b1ba8d7911b6dd5a79e1cebe38558766ec89d8446483d434c7387e330874a70633b2ed5fb8f6a9d8a3401a4abf4
-  data.tar.gz: 49f57c406a2a5c5be83c018c2151ef2e842c266e93d9609dff15684e662bb25580e7c84e2e96a32d1a45094c9baf8ad18f89dbab1e8235127f4d40429f733572
+  metadata.gz: 43a032f78a86df04428af78f32077483b8c01500ebb001ec97100ac3923b4915a28b299cadb8b47fab8fa875c3897732c21eec63769ad98dfd98b901f006b051
+  data.tar.gz: dfff14a83428e2472ff53a50cadf5dd7a270f09523eca4c5aa66e7a3093ca8bd096c9c3d12070ed6a54ef1ddaec4966efd456653e666429f0ce47f04fe68e35a

data/README.md

@@ -157,7 +157,6 @@ All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby.
 
 * Discuss your contribution in an issue
 * Fork it
-* Make your changes, add some tests
-* Ensure all tests pass (`docker-compose -f docker-compose.yml -f docker-compose-ruby-{RUBY_VERSION}.yml run httpx bundle exec rake test`)
+* Make your changes, add some tests (follow the instructions from [here](test/README.md))
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/1_4_1.md

@@ -0,0 +1,19 @@
+# 1.4.1
+
+## Bugfixes
+
+* several `datadog` integration bugfixes
+    * only load the `datadog` integration when the `datadog` sdk is loaded (and not other gems that may define the `Datadog` module, like `dogstatsd`)
+    * do not trace if datadog integration is loaded but disabled
+    * distributed headers are now sent along (when the configuration is enabled, which it is by default)
+* fix for handling multiple `GOAWAY` frames coming from the server (node.js servers seem to send multiple frames on connection timeout)
+* fix regression for when a url is used with `httpx` which is not `http://` or `https://` (should raise `HTTPX::UnsupportedSchemaError`)
+* worked around `IO.copy_stream` which was emitting incorrect bytes for HTTP/2 requests which bodies larger than the maximum supported frame size.
+* multipart requests: make sure that a body declared as `Pathname` is opened for reading in binary mode.
+* `webmock` integration: ensure that request events are emitted (such as plugins and integrations relying in it, such as `datadog` and the OTel integration)
+* native resolver: do not propagate successful name resolutions for connections which were already closed.
+* native resolver: fixed name resolution stalling, in a multi-request to multi-origin scenario, when a resolution timeout would happen.
+
+## Chore
+
+* refactor of the happy eyeballs and connection coalescing logic to not rely on callbacks, and instead on instance variable management (makes code more straightforward to read).

data/doc/release_notes/1_4_2.md

@@ -0,0 +1,20 @@
+# 1.4.2
+
+## Bugfixes
+
+* faraday: use default reason when none is matched by Net::HTTP::STATUS_CODES
+* native resolver: keep sending DNS queries if the socket is available, to avoid busy loops on select
+* native resolver fixes for Happy Eyeballs v2
+  * do not apply resolution delay if the IPv4 IP was not resolved via DNS
+  * ignore ALIAS if DNS response carries IP answers
+  * do not try to query for names already awaiting answer from the resolver
+  * make sure all types of errors are propagated to connections
+  * make sure next candidate is picked up if receiving NX_DOMAIN_NOT_FOUND error from resolver
+* raise error happening before any request is flushed to respective connections (avoids loop on non-actionable selector termination).
+* fix "NoMethodError: undefined method `after' for nil:NilClass", happening for requests flushed into persistent connections which errored, and were retried in a different connection before triggering the timeout callbacks from the previously-closed connection.
+
+
+## Chore
+
+* Refactor of timers to allow for explicit and more performant single timer interval cancellation.
+* default log message restructured to include info about process, thread and caller.

data/doc/release_notes/1_4_3.md

@@ -0,0 +1,11 @@
+# 1.4.3
+
+## Bugfixes
+
+* `webmock` adapter: reassign headers to signature after callbacks are called (these may change the headers before virtual send).
+* do not close request (and its body) right after sending, instead only on response close
+  * prevents retries from failing under the `:retries` plugin
+  * fixes issue when using `faraday-multipart` request bodies
+* retry request with HTTP/1 when receiving an HTTP/2 GOAWAY frame with `HTTP_1_1_REQUIRED` error code.
+* fix wrong method call on HTTP/2 PING frame with unrecognized code.
+* fix EOFError issues on connection termination for long running connections which may have already been terminated by peer and were wrongly trying to complete the HTTP/2 termination handshake.

data/doc/release_notes/1_4_4.md

@@ -0,0 +1,14 @@
+# 1.4.4
+
+## Improvements
+
+* `:stream` plugin: response will now be partially buffered in order to i.e. inspect response status or headers on the response body without buffering the full response
+  * this fixes an issue in the `down` gem integration when used with the `:max_size` option.
+* do not unnecessarily probe for connection liveness if no more requests are inflight, including failed ones.
+* when using persistent connections, do not probe for liveness right after reconnecting after a keep alive timeout.
+
+## Bugfixes
+
+* `:persistent` plugin: do not exhaust retry attempts when probing for (and failing) connection liveness.
+  * since the introduction of per-session connection pools, and consequentially due to the possibility of multiple inactive connections for the same origin being in the pool, which may have been terminated by the peer server, requests would fail before being able to establish a new connection.
+* prevent retrying to connect the TCP socket object when an SSLSocket object is already in place and connecting.

data/lib/httpx/adapters/datadog.rb

@@ -27,62 +27,54 @@ module Datadog::Tracing
       # Enables tracing for httpx requests.
       #
       # A span will be created for each request transaction; the span is created lazily only when
-      # receiving a response, and it is fed the start time stored inside the tracer object.
+      # buffering a request, and it is fed the start time stored inside the tracer object.
       #
       module Plugin
-        class RequestTracer
-          include Contrib::HttpAnnotationHelper
+        module RequestTracer
+          extend Contrib::HttpAnnotationHelper
+
+          module_function
 
           SPAN_REQUEST = "httpx.request"
 
-          # initializes the tracer object on the +request+.
-          def initialize(request)
-            @request = request
-            @start_time = nil
+          # initializes tracing on the +request+.
+          def call(request)
+            return unless configuration(request).enabled
+
+            span = nil
 
             # request objects are reused, when already buffered requests get rerouted to a different
             # connection due to connection issues, or when they already got a response, but need to
             # be retried. In such situations, the original span needs to be extended for the former,
             # while a new is required for the latter.
-            request.on(:idle) { reset }
+            request.on(:idle) do
+              span = nil
+            end
             # the span is initialized when the request is buffered in the parser, which is the closest
             # one gets to actually sending the request.
-            request.on(:headers) { call }
-          end
-
-          # sets up the span start time, while preparing the on response callback.
-          def call(*args)
-            return if @start_time
-
-            start(*args)
+            request.on(:headers) do
+              next if span
 
-            @request.once(:response, &method(:finish))
-          end
+              span = initialize_span(request, now)
+            end
 
-          private
+            request.on(:response) do |response|
+              unless span
+                next unless response.is_a?(::HTTPX::ErrorResponse) && response.error.respond_to?(:connection)
 
-          # just sets the span init time. It can be passed a +start_time+ in cases where
-          # this is collected outside the request transaction.
-          def start(start_time = now)
-            @start_time = start_time
-          end
+                # handles the case when the +error+ happened during name resolution, which means
+                # that the tracing start point hasn't been triggered yet; in such cases, the approximate
+                # initial resolving time is collected from the connection, and used as span start time,
+                # and the tracing object in inserted before the on response callback is called.
+                span = initialize_span(request, response.error.connection.init_time)
 
-          # resets the start time for already finished request transactions.
-          def reset
-            return unless @start_time
+              end
 
-            start
+              finish(response, span)
+            end
           end
 
-          # creates the span from the collected +@start_time+ to what the +response+ state
-          # contains. It also resets internal state to allow this object to be reused.
-          def finish(response)
-            return unless @start_time
-
-            span = initialize_span
-
-            return unless span
-
+          def finish(response, span)
             if response.is_a?(::HTTPX::ErrorResponse)
               span.set_error(response.error)
             else
@@ -92,40 +84,40 @@ module Datadog::Tracing
             end
 
             span.finish
-          ensure
-            @start_time = nil
           end
 
           # return a span initialized with the +@request+ state.
-          def initialize_span
-            verb = @request.verb
-            uri = @request.uri
+          def initialize_span(request, start_time)
+            verb = request.verb
+            uri = request.uri
+
+            config = configuration(request)
 
-            span = create_span(@request)
+            span = create_span(request, config, start_time)
 
             span.resource = verb
 
             # Add additional request specific tags to the span.
 
-            span.set_tag(TAG_URL, @request.path)
+            span.set_tag(TAG_URL, request.path)
             span.set_tag(TAG_METHOD, verb)
 
             span.set_tag(TAG_TARGET_HOST, uri.host)
-            span.set_tag(TAG_TARGET_PORT, uri.port.to_s)
+            span.set_tag(TAG_TARGET_PORT, uri.port)
 
             # Tag as an external peer service
             span.set_tag(TAG_PEER_SERVICE, span.service)
 
-            if configuration[:distributed_tracing]
+            if config[:distributed_tracing]
               propagate_trace_http(
-                Datadog::Tracing.active_trace.to_digest,
-                @request.headers
+                Datadog::Tracing.active_trace,
+                request.headers
               )
             end
 
             # Set analytics sample rate
-            if Contrib::Analytics.enabled?(configuration[:analytics_enabled])
-              Contrib::Analytics.set_sample_rate(span, configuration[:analytics_sample_rate])
+            if Contrib::Analytics.enabled?(config[:analytics_enabled])
+              Contrib::Analytics.set_sample_rate(span, config[:analytics_sample_rate])
             end
 
             span
@@ -138,34 +130,34 @@ module Datadog::Tracing
             ::Datadog::Core::Utils::Time.now.utc
           end
 
-          def configuration
-            @configuration ||= Datadog.configuration.tracing[:httpx, @request.uri.host]
+          def configuration(request)
+            Datadog.configuration.tracing[:httpx, request.uri.host]
           end
 
           if Gem::Version.new(DATADOG_VERSION::STRING) >= Gem::Version.new("2.0.0")
-            def propagate_trace_http(digest, headers)
-              Datadog::Tracing::Contrib::HTTP.inject(digest, headers)
+            def propagate_trace_http(trace, headers)
+              Datadog::Tracing::Contrib::HTTP.inject(trace, headers)
             end
 
-            def create_span(request)
+            def create_span(request, configuration, start_time)
               Datadog::Tracing.trace(
                 SPAN_REQUEST,
-                service: service_name(request.uri.host, configuration, Datadog.configuration_for(self)),
+                service: service_name(request.uri.host, configuration),
                 type: TYPE_OUTBOUND,
-                start_time: @start_time
+                start_time: start_time
               )
             end
           else
-            def propagate_trace_http(digest, headers)
-              Datadog::Tracing::Propagation::HTTP.inject!(digest, headers)
+            def propagate_trace_http(trace, headers)
+              Datadog::Tracing::Propagation::HTTP.inject!(trace.to_digest, headers)
             end
 
-            def create_span(request)
+            def create_span(request, configuration, start_time)
               Datadog::Tracing.trace(
                 SPAN_REQUEST,
-                service: service_name(request.uri.host, configuration, Datadog.configuration_for(self)),
+                service: service_name(request.uri.host, configuration),
                 span_type: TYPE_OUTBOUND,
-                start_time: @start_time
+                start_time: start_time
               )
             end
           end
@@ -178,7 +170,7 @@ module Datadog::Tracing
 
             return unless Datadog::Tracing.enabled?
 
-            RequestTracer.new(self)
+            RequestTracer.call(self)
           end
         end
 
@@ -190,26 +182,6 @@ module Datadog::Tracing
 
             @init_time = ::Datadog::Core::Utils::Time.now.utc
           end
-
-          # handles the case when the +error+ happened during name resolution, which meanns
-          # that the tracing logic hasn't been injected yet; in such cases, the approximate
-          # initial resolving time is collected from the connection, and used as span start time,
-          # and the tracing object in inserted before the on response callback is called.
-          def handle_error(error, request = nil)
-            return super unless Datadog::Tracing.enabled?
-
-            return super unless error.respond_to?(:connection)
-
-            @pending.each do |req|
-              next if request and request == req
-
-              RequestTracer.new(req).call(error.connection.init_time)
-            end
-
-            RequestTracer.new(request).call(error.connection.init_time) if request
-
-            super
-          end
         end
       end
 

data/lib/httpx/adapters/faraday.rb

@@ -149,7 +149,7 @@ module Faraday
 
         module ResponseMethods
           def reason
-            Net::HTTP::STATUS_CODES.fetch(@status)
+            Net::HTTP::STATUS_CODES.fetch(@status, "Non-Standard status code")
           end
         end
       end

data/lib/httpx/adapters/webmock.rb

@@ -58,8 +58,12 @@ module WebMock
           super
 
           connection.once(:unmock_connection) do
+            next unless connection.current_session == self
+
             unless connection.addresses
-              connection.__send__(:callbacks)[:connect_error].clear
+              # reset Happy Eyeballs, fail early
+              connection.sibling = nil
+
               deselect_connection(connection, selector)
             end
             resolve_connection(connection, selector)
@@ -114,8 +118,14 @@ module WebMock
             response = Plugin.build_from_webmock_response(request, mock_response)
             WebMock::CallbackRegistry.invoke_callbacks({ lib: :httpx }, request_signature, mock_response)
             log { "mocking #{request.uri} with #{mock_response.inspect}" }
+            request.transition(:headers)
+            request.transition(:body)
+            request.transition(:trailers)
+            request.transition(:done)
             request.response = response
             request.emit(:response, response)
+            request_signature.headers = request.headers.to_h
+
             response << mock_response.body.dup unless response.is_a?(HTTPX::ErrorResponse)
           elsif WebMock.net_connect_allowed?(request_signature.uri)
             if WebMock::CallbackRegistry.any_callbacks?

data/lib/httpx/callbacks.rb

@@ -4,7 +4,7 @@ module HTTPX
   module Callbacks
     def on(type, &action)
       callbacks(type) << action
-      self
+      action
     end
 
     def once(type, &block)
@@ -12,10 +12,10 @@ module HTTPX
         block.call(*args, &callback)
         :delete
       end
-      self
     end
 
     def emit(type, *args)
+      log { "emit #{type.inspect} callbacks" } if respond_to?(:log)
       callbacks(type).delete_if { |pr| :delete == pr.call(*args) } # rubocop:disable Style/YodaCondition
     end
 

data/lib/httpx/connection/http2.rb

@@ -16,6 +16,12 @@ module HTTPX
       end
     end
 
+    class PingError < Error
+      def initialize
+        super(0, :ping_error)
+      end
+    end
+
     class GoawayError < Error
       def initialize
         super(0, :no_error)
@@ -125,7 +131,7 @@ module HTTPX
     end
 
     def handle_error(ex, request = nil)
-      if ex.instance_of?(TimeoutError) && !@handshake_completed && @connection.state != :closed
+      if ex.is_a?(OperationTimeoutError) && !@handshake_completed && @connection.state != :closed
         @connection.goaway(:settings_timeout, "closing due to settings timeout")
         emit(:close_handshake)
         settings_ex = SettingsTimeoutError.new(ex.timeout, ex.message)
@@ -137,7 +143,7 @@ module HTTPX
 
         emit(:error, req, ex)
       end
-      @pending.each do |req|
+      while (req = @pending.shift)
         next if request && request == req
 
         emit(:error, req, ex)
@@ -311,17 +317,20 @@ module HTTPX
       @streams.delete(request)
 
       if error
-        ex = Error.new(stream.id, error)
-        ex.set_backtrace(caller)
-        response = ErrorResponse.new(request, ex)
-        request.response = response
-        emit(:response, request, response)
+        case error
+        when :http_1_1_required
+          emit(:error, request, error)
+        else
+          ex = Error.new(stream.id, error)
+          ex.set_backtrace(caller)
+          response = ErrorResponse.new(request, ex)
+          request.response = response
+          emit(:response, request, response)
+        end
       else
         response = request.response
         if response && response.is_a?(Response) && response.status == 421
-          ex = MisdirectedRequestError.new(response)
-          ex.set_backtrace(caller)
-          emit(:error, request, ex)
+          emit(:error, request, :http_1_1_required)
         else
           emit(:response, request, response)
         end
@@ -352,7 +361,12 @@ module HTTPX
       is_connection_closed = @connection.state == :closed
       if error
         @buffer.clear if is_connection_closed
-        if error == :no_error
+        case error
+        when :http_1_1_required
+          while (request = @pending.shift)
+            emit(:error, request, error)
+          end
+        when :no_error
           ex = GoawayError.new
           @pending.unshift(*@streams.keys)
           @drains.clear
@@ -360,8 +374,11 @@ module HTTPX
         else
           ex = Error.new(0, error)
         end
-        ex.set_backtrace(caller)
-        handle_error(ex)
+
+        if ex
+          ex.set_backtrace(caller)
+          handle_error(ex)
+        end
       end
       return unless is_connection_closed && @streams.empty?
 
@@ -403,11 +420,9 @@ module HTTPX
     end
 
     def on_pong(ping)
-      if @pings.delete(ping.to_s)
-        emit(:pong)
-      else
-        close(:protocol_error, "ping payload did not match")
-      end
+      raise PingError unless @pings.delete(ping.to_s)
+
+      emit(:pong)
     end
   end
 end