httpx 1.2.6 → 1.4.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0fc8cbe89a9d05f24f615e5de7874c65147954cc8d06524955fea9d0e4ef5598
-  data.tar.gz: e97b5febb579cd1c5e035faef9cf0368ab7148882d2ed95a0454b52fd0d31621
+  metadata.gz: 4aa5b1cfd43d05f50fc7246219d6a90cb5591e11c8bfeee1f1cf62ffa91993fc
+  data.tar.gz: 58f96325ac13b8caddb708464a75ee6dafcfe7b8284f0825803ac3d0c4d8ccc1
 SHA512:
-  metadata.gz: 55631972d465e4ea086dcaced389f7998380f052c369004f45d286d0502c612289dc5a19deff7ca26c9c0cfd6e2c4926ba5e32744c366f18884f97aedb3ad5bf
-  data.tar.gz: 9eb1be90bceb56c335a84b180ae6c7fc065944ae1fd6dab17d9e4802c7fb62c7d3f34d3c98ba41a9cb1b8e503a929eb97cd8a667a9f536090ef266936efb81d9
+  metadata.gz: 43a032f78a86df04428af78f32077483b8c01500ebb001ec97100ac3923b4915a28b299cadb8b47fab8fa875c3897732c21eec63769ad98dfd98b901f006b051
+  data.tar.gz: dfff14a83428e2472ff53a50cadf5dd7a270f09523eca4c5aa66e7a3093ca8bd096c9c3d12070ed6a54ef1ddaec4966efd456653e666429f0ce47f04fe68e35a

data/README.md

@@ -157,7 +157,6 @@ All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby.
 
 * Discuss your contribution in an issue
 * Fork it
-* Make your changes, add some tests
-* Ensure all tests pass (`docker-compose -f docker-compose.yml -f docker-compose-ruby-{RUBY_VERSION}.yml run httpx bundle exec rake test`)
+* Make your changes, add some tests (follow the instructions from [here](test/README.md))
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/1_3_0.md

@@ -0,0 +1,18 @@
+# 1.3.0
+
+## Dependencies
+
+`http-2` v1.0.0 is replacing `http-2-next` as the HTTP/2 parser.
+
+`http-2-next` was forked from `http-2` 5 years ago; its improvements have been merged back to `http-2` recently though, so `http-2-next` willl therefore no longer be maintained.
+
+## Improvements
+
+Request-specific options (`:params`, `:form`, `:json` and `:xml`) are now separately kept by the request, which allows them to share `HTTPX::Options`, and reduce the number of copying / allocations.
+
+This means that `HTTPX::Options` will throw an error if you initialize an object which such keys; this should not happen, as this class is considered internal and you should not be using it directly.
+
+## Fixes
+
+* support for the `datadog` gem v2.0.0 in its adapter has been unblocked, now that the gem has been released.
+* loading the `:cookies` plugin was making the `Session#build_request` private.

data/doc/release_notes/1_3_1.md

@@ -0,0 +1,17 @@
+# 1.3.1
+
+## Improvements
+
+* `:request_timeout` will be applied to all HTTP interactions until the final responses returned to the caller. That includes:
+  * all redirect requests/responses (when using the `:follow_redirects` plugin)
+  * all retried requests/responses (when using the `:retries` plugin)
+  * intermediate requests (such as "100-continue")
+* faraday adapter: allow further plugins of internal session (ex: `builder.adapter(:httpx) { |sess| sess.plugin(:follow_redirects) }...`)
+
+## Bugfixes
+
+* fix connection leak on proxy auth failed (407) handling
+* fix busy loop on deferred requests for the duration interval
+* do not further enqueue deferred requests if they have terminated meanwhile.
+* fix busy loop caused by coalescing connections when one of them is on the DNS resolution phase still.
+* faraday adapter: on parallel mode, skip calling `on_complete` when not defined.

data/doc/release_notes/1_3_2.md

@@ -0,0 +1,6 @@
+# 1.3.2
+
+## Bugfixes
+
+* Prevent `NoMethodError` in an edge case when the `:proxy` plugin is autoloaded via env vars and webmock adapter are used in tandem, and a real request fails.
+* raise invalid uri error if passed request uri does not contain the host part (ex: `"https:/get"`)

data/doc/release_notes/1_3_3.md

@@ -0,0 +1,5 @@
+# 1.3.3
+
+## Bugfixes
+
+* fixing a regression introduced in 1.3.2 associated with the webmock adapter, which expects matchable request bodies to be strings

data/doc/release_notes/1_3_4.md

@@ -0,0 +1,6 @@
+# 1.3.4
+
+## Bugfixes
+
+* webmock adapter: fix tempfile usage in multipart requests.
+* fix: fallback to binary encoding when parsing incoming invalid charset in HTTP "content-type" header.

data/doc/release_notes/1_4_0.md

@@ -0,0 +1,43 @@
+# 1.4.0
+
+## Features
+
+### `:content_digest` plugin
+
+The `:content_digest` can be used to calculate the digest of request payloads and set them in the `"content-digest"` header; it can also validate the integrity of responses which declare the same `"content-digest"` header.
+
+More info under https://honeyryderchuck.gitlab.io/httpx/wiki/Content-Digest
+
+## Per-session connection pools
+
+This architectural changes moves away from per-thread shared connection pools, and into per-session (also thread-safe) connection pools. Unlike before, this enables connections from a session to be reused across threads, as well as limiting the number of connections that can be open on a given origin peer. This fixes long-standing issues, such as reusing connections under a fiber scheduler loop (such as the one from the gem `async`).
+
+A new `:pool_options` option is introduced, which can be passed an hash with the following sub-options:
+
+* `:max_connections_per_origin`: maximum number of connections a pool allows (unbounded by default, for backwards compatibility).
+* `:pool_timeout`: the number of seconds a session will wait for a connection to be checked out (default: 5)
+
+More info under https://honeyryderchuck.gitlab.io/httpx/wiki/Connection-Pools
+
+
+## Improvements
+
+* `:aws_sigv4` plugin: improved digest calculation on compressed request bodies by buffering content to a tempfile.
+* `HTTPX::Response#json` will parse payload from extended json MIME types (like `application/ld+json`, `application/hal+json`, ...).
+
+## Bugfixes
+
+* `:aws_sigv4` plugin: do not try to rewind a request body which yields chunks.
+* fixed request encoding when `:json` param is passed, and the `oj` gem is used (by using the `:compat` flag).
+* native resolver: on message truncation, bubble up tcp handshake errors as resolve errors.
+* allow `HTTPX::Response#json` to accept extended JSON mime types (such as responses with `content-type: application/ld+json`)
+
+## Chore
+
+* default options are now fully frozen (in case anyone relies on overriding them).
+
+### `:xml` plugin
+
+XML encoding/decoding (via `:xml` request param, and `HTTPX::Response#xml`) is now available via the `:xml` plugin.
+
+Using `HTTPX::Response#xml` without the plugin will issue a deprecation warning.

data/doc/release_notes/1_4_1.md

@@ -0,0 +1,19 @@
+# 1.4.1
+
+## Bugfixes
+
+* several `datadog` integration bugfixes
+    * only load the `datadog` integration when the `datadog` sdk is loaded (and not other gems that may define the `Datadog` module, like `dogstatsd`)
+    * do not trace if datadog integration is loaded but disabled
+    * distributed headers are now sent along (when the configuration is enabled, which it is by default)
+* fix for handling multiple `GOAWAY` frames coming from the server (node.js servers seem to send multiple frames on connection timeout)
+* fix regression for when a url is used with `httpx` which is not `http://` or `https://` (should raise `HTTPX::UnsupportedSchemaError`)
+* worked around `IO.copy_stream` which was emitting incorrect bytes for HTTP/2 requests which bodies larger than the maximum supported frame size.
+* multipart requests: make sure that a body declared as `Pathname` is opened for reading in binary mode.
+* `webmock` integration: ensure that request events are emitted (such as plugins and integrations relying in it, such as `datadog` and the OTel integration)
+* native resolver: do not propagate successful name resolutions for connections which were already closed.
+* native resolver: fixed name resolution stalling, in a multi-request to multi-origin scenario, when a resolution timeout would happen.
+
+## Chore
+
+* refactor of the happy eyeballs and connection coalescing logic to not rely on callbacks, and instead on instance variable management (makes code more straightforward to read).

data/doc/release_notes/1_4_2.md

@@ -0,0 +1,20 @@
+# 1.4.2
+
+## Bugfixes
+
+* faraday: use default reason when none is matched by Net::HTTP::STATUS_CODES
+* native resolver: keep sending DNS queries if the socket is available, to avoid busy loops on select
+* native resolver fixes for Happy Eyeballs v2
+  * do not apply resolution delay if the IPv4 IP was not resolved via DNS
+  * ignore ALIAS if DNS response carries IP answers
+  * do not try to query for names already awaiting answer from the resolver
+  * make sure all types of errors are propagated to connections
+  * make sure next candidate is picked up if receiving NX_DOMAIN_NOT_FOUND error from resolver
+* raise error happening before any request is flushed to respective connections (avoids loop on non-actionable selector termination).
+* fix "NoMethodError: undefined method `after' for nil:NilClass", happening for requests flushed into persistent connections which errored, and were retried in a different connection before triggering the timeout callbacks from the previously-closed connection.
+
+
+## Chore
+
+* Refactor of timers to allow for explicit and more performant single timer interval cancellation.
+* default log message restructured to include info about process, thread and caller.

data/doc/release_notes/1_4_3.md

@@ -0,0 +1,11 @@
+# 1.4.3
+
+## Bugfixes
+
+* `webmock` adapter: reassign headers to signature after callbacks are called (these may change the headers before virtual send).
+* do not close request (and its body) right after sending, instead only on response close
+  * prevents retries from failing under the `:retries` plugin
+  * fixes issue when using `faraday-multipart` request bodies
+* retry request with HTTP/1 when receiving an HTTP/2 GOAWAY frame with `HTTP_1_1_REQUIRED` error code.
+* fix wrong method call on HTTP/2 PING frame with unrecognized code.
+* fix EOFError issues on connection termination for long running connections which may have already been terminated by peer and were wrongly trying to complete the HTTP/2 termination handshake.

data/doc/release_notes/1_4_4.md

@@ -0,0 +1,14 @@
+# 1.4.4
+
+## Improvements
+
+* `:stream` plugin: response will now be partially buffered in order to i.e. inspect response status or headers on the response body without buffering the full response
+  * this fixes an issue in the `down` gem integration when used with the `:max_size` option.
+* do not unnecessarily probe for connection liveness if no more requests are inflight, including failed ones.
+* when using persistent connections, do not probe for liveness right after reconnecting after a keep alive timeout.
+
+## Bugfixes
+
+* `:persistent` plugin: do not exhaust retry attempts when probing for (and failing) connection liveness.
+  * since the introduction of per-session connection pools, and consequentially due to the possibility of multiple inactive connections for the same origin being in the pool, which may have been terminated by the peer server, requests would fail before being able to establish a new connection.
+* prevent retrying to connect the TCP socket object when an SSLSocket object is already in place and connecting.

data/lib/httpx/adapters/datadog.rb

@@ -27,62 +27,54 @@ module Datadog::Tracing
       # Enables tracing for httpx requests.
       #
       # A span will be created for each request transaction; the span is created lazily only when
-      # receiving a response, and it is fed the start time stored inside the tracer object.
+      # buffering a request, and it is fed the start time stored inside the tracer object.
       #
       module Plugin
-        class RequestTracer
-          include Contrib::HttpAnnotationHelper
+        module RequestTracer
+          extend Contrib::HttpAnnotationHelper
+
+          module_function
 
           SPAN_REQUEST = "httpx.request"
 
-          # initializes the tracer object on the +request+.
-          def initialize(request)
-            @request = request
-            @start_time = nil
+          # initializes tracing on the +request+.
+          def call(request)
+            return unless configuration(request).enabled
+
+            span = nil
 
             # request objects are reused, when already buffered requests get rerouted to a different
             # connection due to connection issues, or when they already got a response, but need to
             # be retried. In such situations, the original span needs to be extended for the former,
             # while a new is required for the latter.
-            request.on(:idle) { reset }
+            request.on(:idle) do
+              span = nil
+            end
             # the span is initialized when the request is buffered in the parser, which is the closest
             # one gets to actually sending the request.
-            request.on(:headers) { call }
-          end
-
-          # sets up the span start time, while preparing the on response callback.
-          def call(*args)
-            return if @start_time
-
-            start(*args)
+            request.on(:headers) do
+              next if span
 
-            @request.once(:response, &method(:finish))
-          end
+              span = initialize_span(request, now)
+            end
 
-          private
+            request.on(:response) do |response|
+              unless span
+                next unless response.is_a?(::HTTPX::ErrorResponse) && response.error.respond_to?(:connection)
 
-          # just sets the span init time. It can be passed a +start_time+ in cases where
-          # this is collected outside the request transaction.
-          def start(start_time = now)
-            @start_time = start_time
-          end
+                # handles the case when the +error+ happened during name resolution, which means
+                # that the tracing start point hasn't been triggered yet; in such cases, the approximate
+                # initial resolving time is collected from the connection, and used as span start time,
+                # and the tracing object in inserted before the on response callback is called.
+                span = initialize_span(request, response.error.connection.init_time)
 
-          # resets the start time for already finished request transactions.
-          def reset
-            return unless @start_time
+              end
 
-            start
+              finish(response, span)
+            end
           end
 
-          # creates the span from the collected +@start_time+ to what the +response+ state
-          # contains. It also resets internal state to allow this object to be reused.
-          def finish(response)
-            return unless @start_time
-
-            span = initialize_span
-
-            return unless span
-
+          def finish(response, span)
             if response.is_a?(::HTTPX::ErrorResponse)
               span.set_error(response.error)
             else
@@ -92,40 +84,40 @@ module Datadog::Tracing
             end
 
             span.finish
-          ensure
-            @start_time = nil
           end
 
           # return a span initialized with the +@request+ state.
-          def initialize_span
-            verb = @request.verb
-            uri = @request.uri
+          def initialize_span(request, start_time)
+            verb = request.verb
+            uri = request.uri
 
-            span = create_span(@request)
+            config = configuration(request)
+
+            span = create_span(request, config, start_time)
 
             span.resource = verb
 
             # Add additional request specific tags to the span.
 
-            span.set_tag(TAG_URL, @request.path)
+            span.set_tag(TAG_URL, request.path)
             span.set_tag(TAG_METHOD, verb)
 
             span.set_tag(TAG_TARGET_HOST, uri.host)
-            span.set_tag(TAG_TARGET_PORT, uri.port.to_s)
+            span.set_tag(TAG_TARGET_PORT, uri.port)
 
             # Tag as an external peer service
             span.set_tag(TAG_PEER_SERVICE, span.service)
 
-            if configuration[:distributed_tracing]
+            if config[:distributed_tracing]
               propagate_trace_http(
-                Datadog::Tracing.active_trace.to_digest,
-                @request.headers
+                Datadog::Tracing.active_trace,
+                request.headers
               )
             end
 
             # Set analytics sample rate
-            if Contrib::Analytics.enabled?(configuration[:analytics_enabled])
-              Contrib::Analytics.set_sample_rate(span, configuration[:analytics_sample_rate])
+            if Contrib::Analytics.enabled?(config[:analytics_enabled])
+              Contrib::Analytics.set_sample_rate(span, config[:analytics_sample_rate])
             end
 
             span
@@ -138,34 +130,34 @@ module Datadog::Tracing
             ::Datadog::Core::Utils::Time.now.utc
           end
 
-          def configuration
-            @configuration ||= Datadog.configuration.tracing[:httpx, @request.uri.host]
+          def configuration(request)
+            Datadog.configuration.tracing[:httpx, request.uri.host]
           end
 
-          if Gem::Version.new(DATADOG_VERSION::STRING) >= Gem::Version.new("2.0.0.beta1")
-            def propagate_trace_http(digest, headers)
-              Datadog::Tracing::Contrib::HTTP.inject(digest, headers)
+          if Gem::Version.new(DATADOG_VERSION::STRING) >= Gem::Version.new("2.0.0")
+            def propagate_trace_http(trace, headers)
+              Datadog::Tracing::Contrib::HTTP.inject(trace, headers)
             end
 
-            def create_span(request)
+            def create_span(request, configuration, start_time)
               Datadog::Tracing.trace(
                 SPAN_REQUEST,
-                service: service_name(request.uri.host, configuration, Datadog.configuration_for(self)),
+                service: service_name(request.uri.host, configuration),
                 type: TYPE_OUTBOUND,
-                start_time: @start_time
+                start_time: start_time
               )
             end
           else
-            def propagate_trace_http(digest, headers)
-              Datadog::Tracing::Propagation::HTTP.inject!(digest, headers)
+            def propagate_trace_http(trace, headers)
+              Datadog::Tracing::Propagation::HTTP.inject!(trace.to_digest, headers)
             end
 
-            def create_span(request)
+            def create_span(request, configuration, start_time)
               Datadog::Tracing.trace(
                 SPAN_REQUEST,
-                service: service_name(request.uri.host, configuration, Datadog.configuration_for(self)),
+                service: service_name(request.uri.host, configuration),
                 span_type: TYPE_OUTBOUND,
-                start_time: @start_time
+                start_time: start_time
               )
             end
           end
@@ -178,7 +170,7 @@ module Datadog::Tracing
 
             return unless Datadog::Tracing.enabled?
 
-            RequestTracer.new(self)
+            RequestTracer.call(self)
           end
         end
 
@@ -190,22 +182,6 @@ module Datadog::Tracing
 
             @init_time = ::Datadog::Core::Utils::Time.now.utc
           end
-
-          # handles the case when the +error+ happened during name resolution, which meanns
-          # that the tracing logic hasn't been injected yet; in such cases, the approximate
-          # initial resolving time is collected from the connection, and used as span start time,
-          # and the tracing object in inserted before the on response callback is called.
-          def handle_error(error)
-            return super unless Datadog::Tracing.enabled?
-
-            return super unless error.respond_to?(:connection)
-
-            @pending.each do |request|
-              RequestTracer.new(request).call(error.connection.init_time)
-            end
-
-            super
-          end
         end
       end
 

data/lib/httpx/adapters/faraday.rb

@@ -30,6 +30,7 @@ module Faraday
           end
           @connection = @connection.plugin(OnDataPlugin) if env.request.stream_response?
 
+          @connection = @config_block.call(@connection) || @connection if @config_block
           @connection
         end
 
@@ -107,9 +108,11 @@ module Faraday
             ssl_options
           end
         else
+          # :nocov:
           def ssl_options_from_env(*)
             {}
           end
+          # :nocov:
         end
       end
 
@@ -146,7 +149,7 @@ module Faraday
 
         module ResponseMethods
           def reason
-            Net::HTTP::STATUS_CODES.fetch(@status)
+            Net::HTTP::STATUS_CODES.fetch(@status, "Non-Standard status code")
           end
         end
       end
@@ -212,7 +215,7 @@ module Faraday
             Array(responses).each_with_index do |response, index|
               handler = @handlers[index]
               handler.on_response.call(response)
-              handler.on_complete.call(handler.env)
+              handler.on_complete.call(handler.env) if handler.on_complete
             end
           end
         rescue ::HTTPX::TimeoutError => e

data/lib/httpx/adapters/webmock.rb

@@ -20,7 +20,7 @@ module WebMock
           WebMock::RequestSignature.new(
             request.verb.downcase.to_sym,
             uri.to_s,
-            body: request.body.each.to_a.join,
+            body: request.body.to_s,
             headers: request.headers.to_h
           )
         end
@@ -47,21 +47,27 @@ module WebMock
         end
 
         def build_error_response(request, exception)
-          HTTPX::ErrorResponse.new(request, exception, request.options)
+          HTTPX::ErrorResponse.new(request, exception)
         end
       end
 
       module InstanceMethods
-        def init_connection(*)
-          connection = super
+        private
+
+        def do_init_connection(connection, selector)
+          super
+
           connection.once(:unmock_connection) do
+            next unless connection.current_session == self
+
             unless connection.addresses
-              connection.__send__(:callbacks)[:connect_error].clear
-              pool.__send__(:unregister_connection, connection)
+              # reset Happy Eyeballs, fail early
+              connection.sibling = nil
+
+              deselect_connection(connection, selector)
             end
-            pool.__send__(:resolve_connection, connection)
+            resolve_connection(connection, selector)
           end
-          connection
         end
       end
 
@@ -100,6 +106,10 @@ module WebMock
           super
         end
 
+        def terminate
+          force_reset
+        end
+
         def send(request)
           request_signature = Plugin.build_webmock_request_signature(request)
           WebMock::RequestRegistry.instance.requested_signatures.put(request_signature)
@@ -108,8 +118,14 @@ module WebMock
             response = Plugin.build_from_webmock_response(request, mock_response)
             WebMock::CallbackRegistry.invoke_callbacks({ lib: :httpx }, request_signature, mock_response)
             log { "mocking #{request.uri} with #{mock_response.inspect}" }
+            request.transition(:headers)
+            request.transition(:body)
+            request.transition(:trailers)
+            request.transition(:done)
             request.response = response
             request.emit(:response, response)
+            request_signature.headers = request.headers.to_h
+
             response << mock_response.body.dup unless response.is_a?(HTTPX::ErrorResponse)
           elsif WebMock.net_connect_allowed?(request_signature.uri)
             if WebMock::CallbackRegistry.any_callbacks?

data/lib/httpx/callbacks.rb

@@ -4,7 +4,7 @@ module HTTPX
   module Callbacks
     def on(type, &action)
       callbacks(type) << action
-      self
+      action
     end
 
     def once(type, &block)
@@ -12,15 +12,10 @@ module HTTPX
         block.call(*args, &callback)
         :delete
       end
-      self
-    end
-
-    def only(type, &block)
-      callbacks(type).clear
-      on(type, &block)
     end
 
     def emit(type, *args)
+      log { "emit #{type.inspect} callbacks" } if respond_to?(:log)
       callbacks(type).delete_if { |pr| :delete == pr.call(*args) } # rubocop:disable Style/YodaCondition
     end
 

data/lib/httpx/chainable.rb

@@ -73,7 +73,7 @@ module HTTPX
         ].include?(callback)
 
         warn "DEPRECATION WARNING: calling `.#{meth}` on plain HTTPX sessions is deprecated. " \
-             "Use HTTPX.plugin(:callbacks).#{meth} instead."
+             "Use `HTTPX.plugin(:callbacks).#{meth}` instead."
 
         plugin(:callbacks).__send__(meth, *args, **options, &blk)
       else
@@ -101,4 +101,6 @@ module HTTPX
       end
     end
   end
+
+  extend Chainable
 end

data/lib/httpx/connection/http1.rb

@@ -15,7 +15,7 @@ module HTTPX
     attr_accessor :max_concurrent_requests
 
     def initialize(buffer, options)
-      @options = Options.new(options)
+      @options = options
       @max_concurrent_requests = @options.max_concurrent_requests || MAX_REQUESTS
       @max_requests = @options.max_requests
       @parser = Parser::HTTP1.new(self)
@@ -146,7 +146,7 @@ module HTTPX
 
       response << chunk
     rescue StandardError => e
-      error_response = ErrorResponse.new(request, e, request.options)
+      error_response = ErrorResponse.new(request, e)
       request.response = error_response
       dispatch
     end
@@ -197,7 +197,7 @@ module HTTPX
       end
     end
 
-    def handle_error(ex)
+    def handle_error(ex, request = nil)
       if (ex.is_a?(EOFError) || ex.is_a?(TimeoutError)) && @request && @request.response &&
          !@request.response.headers.key?("content-length") &&
          !@request.response.headers.key?("transfer-encoding")
@@ -211,11 +211,15 @@ module HTTPX
       if @pipelining
         catch(:called) { disable }
       else
-        @requests.each do |request|
-          emit(:error, request, ex)
+        @requests.each do |req|
+          next if request && request == req
+
+          emit(:error, req, ex)
         end
-        @pending.each do |request|
-          emit(:error, request, ex)
+        @pending.each do |req|
+          next if request && request == req
+
+          emit(:error, req, ex)
         end
       end
     end