httpx 1.7.2 → 1.7.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c67e4695d8ef368321f14a3c113daad0e49280413e3da42272644ad84fe6f622
-  data.tar.gz: 62bb1ab9d91ca69c9b1fa561051ffb0ed137afb53e0b40f71cfd477a84a78773
+  metadata.gz: 125df778093197c9a1fe8872211cf4085a5a885a40536b58501ce1938f8d28ee
+  data.tar.gz: 5a17b5b02212f65ba3472f4dfda96a017fc1ef9a4fb7a3dce85d7b4ac6a8c6bb
 SHA512:
-  metadata.gz: 266576dae6b8ed604228b464281239b760df9682d27e9010b18ac7dbab4fb00f23115e5bcae1a0ecee6b191ee03a0eca758c37f8f0baee0e2ad9df5ca0c7eff6
-  data.tar.gz: e355df3c634811d0d8e08bcb10b661a4f486073e6ca808ca3af3585a4d014cd404c8d7a55e38d0d05419f205d9dc81a54fb5a17eab7e1716c38c160297963e0f
+  metadata.gz: b6a4014db970d25a1ac51ea73ac5bd828bf8c0f1012ec778abdf054d9dd6dcfe6cf7fc5f73f915a39d74e4e420329e1293e020d63955601e6f669ae6164a016b
+  data.tar.gz: 96b98469fe6194bbd3427adfd1bfc6032be9c44f2c68ff301a6da20a90741b1979aa194c113a11e1b82cb9a54c3559ef7c35b908d41c10297746e438b557eca3

data/README.md

@@ -46,7 +46,9 @@ And that's the simplest one there is. But you can also do:
 HTTPX.post("http://example.com", form: { user: "john", password: "pass" })
 
 http = HTTPX.with(headers: { "x-my-name" => "joe" })
-http.patch("http://example.com/file", body: File.open("path/to/file")) # request body is streamed
+File.open("path/to/file") do |file|
+  http.patch("http://example.com/file", body: file) # request body is streamed
+end
 ```
 
 If you want to do some more things with the response, you can get an `HTTPX::Response`:

data/doc/release_notes/1_7_3.md

@@ -0,0 +1,29 @@
+# 1.7.3
+
+## Improvements
+
+### cookies plugin: Jar as CookieStore
+
+While previously an implementation detail, the cookie jar from a `:cookie` plugin-enabled session can now be manipulated by the end user:
+
+```ruby
+cookies_sess = HTTPX.plugin(:cookies)
+
+jar = cookies.make_jar
+
+sess = cookies_ses.with(cookies: jar)
+
+# perform requests using sess, get/set/delete cookies in jar
+```
+
+The jar API now closely follows the [Web Cookie Store API](https://developer.mozilla.org/en-US/docs/Web/API/CookieStore), by providing the same set of functions.
+
+Some API backwards compatibility is maintained, however since this was an internal implementation detail, this effort isn't meant to be thorough.
+
+## Bugfixes
+
+* `http-2`: clear buffered data chunks when receiving a `GOAWAY` stream frame; without this, the client kept sending the corresponding `DATA` frames, despite the peer server making it known that it wouldn't process it. While this is valid HTTP/2, this could increase the connection window until a point where it'd go over the max frame size. this issue was observed during large file uploads where the first request could fail and make the client renegotiate.
+* `webmock` adapter: fixed response body length accounting which was making `response.body.empty?` return true for responses with payload.
+* `:rate_limiter` plugin relies on an internal refactoring to be able to wait for the time suggested by the peer server instead of the potentially relying on custom user logic via own `:retry_after`.
+* `:fiber_concurrency`: fix wrong names for native/system resolver overrides.
+* connection: fix for race condition when closing the connection, where the state only transitions to `closed` after checking the connection back in to the pool, potentially corrupting it if another session meanwhile has picked it up and manipulated it.

data/doc/release_notes/1_7_4.md

@@ -0,0 +1,42 @@
+# 1.7.4
+
+## Features
+
+### Tracing plugin
+
+A new `:tracing` plugin was introduced. It adds support for a new option, `:tracer`, which accepts an object which responds to the following callbacks:
+
+* `#enabled?(request)` - should return true or false depending on whether tracing is enabled
+* `#start(request)` - called when a request is about to be sent
+* `#finish(request, response)` - called when a response is received
+* `#reset(request)` - called when a request is being prepared to be resent, in cases where it makes sense (i.e. when a request is retried).
+
+You can pass chain several tracers, and callbacks will be relayed to all of them:
+
+```ruby
+HTTP.plugin(:tracing).with(tracer: telemetry_platform_tracer).with(tracer: telemetry2_platform_tracer)
+```
+
+This was developed to be the foundation on top of which the datadag and OTel integrations will be built.
+
+## Improvements
+
+* try fetching response immediately after send the request to the connection; this allows returning from errors much earlier and bug free than doing another round of waits on I/O.
+* when a connection is reconnected, and it was established the first time that the peer can accept only 1 request at a time, the connection will keep that informaation and keep sending requests 1 at a time afterwards.
+
+## Bugfixes
+
+* fix regression from introducing connection post state transition callbacks, by foregoing disconnect when there's pending backlog.
+* transition requests to `:idle` before routing them to a different connection on merge (this could possibly leave dangling timeout callbacks otherwise).
+* `:brotli` plugin was integrated with the stream writer component which allows writing compressed payload in chunks.
+* `:brotli` plugin integrates with the `brotli` gem v0.8.0, which fixed an issue dealing with large payload responses due to the lack of support for decoding payloads in chunks.
+* http1 parser: reset before early returning on `Upgrade` responses (it was left in an invalid "parsing headers", which in the case of a keep-alive connection, would cause the next request to fail being parsed).
+* `datadog` adapter: fixed initialization of the request start time after connections were opened (it was being set to connection initialization time every time, instead of just on the first request before connection is established).
+* parsers: also reroute non-completed in-flight requests back to the connection so they can be retried (previously, only pending requests were).
+* `:proxy` plugin: do not try disconnecting unnecessarily when resetting may already do so (if conditions apply).
+* `:proxy` plugin: removed call to unexisting `#reset!` function.
+* `:proxy` plugin: also close wrapped sockets.
+* connection: on force_close, move connection disconnection logic below so that, if requests are reenqueued from the parser, this can be halted.
+* connection: when transition to `:idle`, reenqueue requests from parser before resetting it.
+* implement `#lazy_resolve` on resolvers, as when they're picked from the selector (instead of from the pool), they may not be wrapped by a Multi proxy.
+* allow resolvers transitioning from `:idle` to `:closed` and forego disconnecting when the resolver is not able to transition to `:closed` (which protects from a possible fiber scheduler context switch which changed the state under the hood).

data/lib/httpx/adapters/datadog.rb

@@ -46,35 +46,25 @@ module Datadog::Tracing
 
           SPAN_REQUEST = "httpx.request"
 
-          # initializes tracing on the +request+.
-          def call(request)
-            return unless configuration(request).enabled
-
-            span = nil
-
-            # request objects are reused, when already buffered requests get rerouted to a different
-            # connection due to connection issues, or when they already got a response, but need to
-            # be retried. In such situations, the original span needs to be extended for the former,
-            # while a new is required for the latter.
-            request.on(:idle) do
-              span = nil
-            end
-            # the span is initialized when the request is buffered in the parser, which is the closest
-            # one gets to actually sending the request.
-            request.on(:headers) do
-              next if span
+          def enabled?(request)
+            configuration(request).enabled
+          end
 
-              span = initialize_span(request, now)
-            end
+          def start(request)
+            request.datadog_span = initialize_span(request, now)
+          end
+
+          def reset(request)
+            request.datadog_span = nil
+          end
 
-            request.on(:response) do |response|
-              span = initialize_span(request, request.init_time) if !span && request.init_time
+          def finish(request, response)
+            request.datadog_span ||= initialize_span(request, request.init_time) if request.init_time
 
-              finish(response, span)
-            end
+            finish_span(response, request.datadog_span)
           end
 
-          def finish(response, span)
+          def finish_span(response, span)
             if response.is_a?(::HTTPX::ErrorResponse)
               span.set_error(response.error)
             else
@@ -137,9 +127,9 @@ module Datadog::Tracing
             ) if Datadog.configuration.tracing.respond_to?(:header_tags)
 
             span
-          rescue StandardError => e
-            Datadog.logger.error("error preparing span for http request: #{e}")
-            Datadog.logger.error(e.backtrace)
+        rescue StandardError => e
+          Datadog.logger.error("error preparing span for http request: #{e}")
+          Datadog.logger.error(e.backtrace)
           end
 
           def now
@@ -179,44 +169,18 @@ module Datadog::Tracing
           end
         end
 
-        module RequestMethods
-          attr_accessor :init_time
-
-          # intercepts request initialization to inject the tracing logic.
-          def initialize(*)
-            super
-
-            @init_time = nil
-
-            return unless Datadog::Tracing.enabled?
-
-            RequestTracer.call(self)
+        class << self
+          def load_dependencies(klass)
+            klass.plugin(:tracing)
           end
 
-          def response=(*)
-            # init_time should be set when it's send to a connection.
-            # However, there are situations where connection initialization fails.
-            # Example is the :ssrf_filter plugin, which raises an error on
-            # initialize if the host is an IP which matches against the known set.
-            # in such cases, we'll just set here right here.
-            @init_time ||= ::Datadog::Core::Utils::Time.now.utc
-
-            super
+          def extra_options(options)
+            options.merge(tracer: RequestTracer)
           end
         end
 
-        module ConnectionMethods
-          def initialize(*)
-            super
-
-            @init_time = ::Datadog::Core::Utils::Time.now.utc
-          end
-
-          def send(request)
-            request.init_time ||= @init_time
-
-            super
-          end
+        module RequestMethods
+          attr_accessor :datadog_span
         end
       end
 

data/lib/httpx/adapters/webmock.rb

@@ -82,6 +82,7 @@ module WebMock
 
         def mock!
           @mocked = true
+          @body.mock!
         end
 
         def mocked?
@@ -90,10 +91,8 @@ module WebMock
       end
 
       module ResponseBodyMethods
-        def decode_chunk(chunk)
-          return chunk if @response.mocked?
-
-          super
+        def mock!
+          @inflaters = nil
         end
       end
 

data/lib/httpx/connection/http1.rb

@@ -49,7 +49,12 @@ module HTTPX
       @max_requests = @options.max_requests || MAX_REQUESTS
       @parser.reset!
       @handshake_completed = false
+      reset_requests
+    end
+
+    def reset_requests
       @pending.unshift(*@requests)
+      @requests.clear
     end
 
     def close
@@ -175,6 +180,7 @@ module HTTPX
 
       if @parser.upgrade?
         response << @parser.upgrade_data
+        @parser.reset!
         throw(:called)
       end
 
@@ -280,7 +286,6 @@ module HTTPX
     end
 
     def disable_pipelining
-      return if @requests.empty?
       # do not disable pipelining if already set to 1 request at a time
       return if @max_concurrent_requests == 1
 

data/lib/httpx/connection/http2.rb

@@ -3,8 +3,6 @@
 require "securerandom"
 require "http/2"
 
-HTTP2::Connection.__send__(:public, :send_buffer) if HTTP2::VERSION < "1.1.1"
-
 module HTTPX
   class Connection::HTTP2
     include Callbacks
@@ -163,6 +161,8 @@ module HTTPX
       @pings.any?
     end
 
+    def reset_requests; end
+
     private
 
     def can_buffer_more_requests?
@@ -215,9 +215,7 @@ module HTTPX
     def handle_stream(stream, request)
       request.on(:refuse, &method(:on_stream_refuse).curry(3)[stream, request])
       stream.on(:close, &method(:on_stream_close).curry(3)[stream, request])
-      stream.on(:half_close) do
-        log(level: 2) { "#{stream.id}: waiting for response..." }
-      end
+      stream.on(:half_close) { on_stream_half_close(stream, request) }
       stream.on(:altsvc, &method(:on_altsvc).curry(2)[request.origin])
       stream.on(:headers, &method(:on_stream_headers).curry(3)[stream, request])
       stream.on(:data, &method(:on_stream_data).curry(3)[stream, request])
@@ -302,7 +300,7 @@ module HTTPX
       end
 
       log(color: :yellow) do
-        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{log_redact_headers(v)}" }.join("\n")
+        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{k == ":status" ? v : log_redact_headers(v)}" }.join("\n")
       end
       _, status = h.shift
       headers = request.options.headers_class.new(h)
@@ -331,6 +329,16 @@ module HTTPX
       stream.close
     end
 
+    def on_stream_half_close(stream, _request)
+      unless stream.send_buffer.empty?
+        stream.send_buffer.clear
+        stream.data("", end_stream: true)
+      end
+
+      # TODO: omit log line if response already here
+      log(level: 2) { "#{stream.id}: waiting for response..." }
+    end
+
     def on_stream_close(stream, request, error)
       return if error == :stream_closed && !@streams.key?(request)
 
@@ -404,34 +412,39 @@ module HTTPX
 
     def on_frame_sent(frame)
       log(level: 2) { "#{frame[:stream]}: frame was sent!" }
-      log(level: 2, color: :blue) do
-        payload =
-          case frame[:type]
-          when :data
-            frame.merge(payload: frame[:payload].bytesize)
-          when :headers, :ping
-            frame.merge(payload: log_redact_headers(frame[:payload]))
-          else
-            frame
-          end
-        "#{frame[:stream]}: #{payload}"
-      end
+      log(level: 2, color: :blue) { "#{frame[:stream]}: #{frame_with_extra_info(frame)}" }
     end
 
     def on_frame_received(frame)
       log(level: 2) { "#{frame[:stream]}: frame was received!" }
-      log(level: 2, color: :magenta) do
-        payload =
-          case frame[:type]
-          when :data
-            frame.merge(payload: frame[:payload].bytesize)
-          when :headers, :ping
-            frame.merge(payload: log_redact_headers(frame[:payload]))
-          else
-            frame
-          end
-        "#{frame[:stream]}: #{payload}"
-      end
+      log(level: 2, color: :magenta) { "#{frame[:stream]}: #{frame_with_extra_info(frame)}" }
+    end
+
+    def frame_with_extra_info(frame)
+      case frame[:type]
+      when :data
+        frame.merge(payload: frame[:payload].bytesize)
+      when :headers, :ping
+        frame.merge(payload: log_redact_headers(frame[:payload]))
+      when :window_update
+        connection_or_stream = if (id = frame[:stream]).zero?
+          @connection
+        else
+          @streams.each_value.find { |s| s.id == id }
+        end
+        if connection_or_stream
+          frame.merge(
+            local_window: connection_or_stream.local_window,
+            remote_window: connection_or_stream.remote_window,
+            buffered_amount: connection_or_stream.buffered_amount,
+            stream_state: connection_or_stream.state,
+          )
+        else
+          frame
+        end
+      else
+        frame
+      end.merge(connection_state: @connection.state)
     end
 
     def on_altsvc(origin, frame)

data/lib/httpx/connection.rb

@@ -45,10 +45,10 @@ module HTTPX
     protected :ssl_session, :sibling
 
     def initialize(uri, options)
-      @current_session = @current_selector =
-        @parser = @sibling = @coalesced_connection = @altsvc_connection =
-                               @family = @io = @ssl_session = @timeout =
-                                                 @connected_at = @response_received_at = nil
+      @current_session = @current_selector = @max_concurrent_requests =
+                           @parser = @sibling = @coalesced_connection = @altsvc_connection =
+                                                  @family = @io = @ssl_session = @timeout =
+                                                                    @connected_at = @response_received_at = nil
 
       @exhausted = @cloned = @main_sibling = false
 
@@ -154,6 +154,7 @@ module HTTPX
         end if @io
       end
       connection.purge_pending do |req|
+        req.transition(:idle)
         send(req)
       end
     end
@@ -161,8 +162,9 @@ module HTTPX
     def purge_pending(&block)
       pendings = []
       if @parser
-        @inflight -= @parser.pending.size
-        pendings << @parser.pending
+        pending = @parser.pending
+        @inflight -= pending.size
+        pendings << pending
       end
       pendings << @pending
       pendings.each do |pending|
@@ -227,6 +229,9 @@ module HTTPX
         consume
       end
       nil
+    rescue IOError => e
+      @write_buffer.clear
+      on_io_error(e)
     rescue StandardError => e
       @write_buffer.clear
       on_error(e)
@@ -256,16 +261,17 @@ module HTTPX
     # bypasses state machine rules while setting the connection in the
     # :closed state.
     def force_close(delete_pending = false)
+      force_purge
+      return unless @state == :closed
+
       if delete_pending
         @pending.clear
       elsif (parser = @parser)
         enqueue_pending_requests_from_parser(parser)
       end
-      return if @state == :closed
 
-      @state = :closed
-      @write_buffer.clear
-      purge_after_closed
+      return unless @pending.empty?
+
       disconnect
       emit(:force_closed, delete_pending)
     end
@@ -281,6 +287,15 @@ module HTTPX
     def reset
       return if @state == :closing || @state == :closed
 
+      parser = @parser
+
+      if parser && parser.respond_to?(:max_concurrent_requests)
+        # if connection being reset has at some downgraded the number of concurrent
+        # requests, such as in the case where an attempt to use HTTP/1 pipelining failed,
+        # keep that information around.
+        @max_concurrent_requests = parser.max_concurrent_requests
+      end
+
       transition(:closing)
 
       transition(:closed)
@@ -323,7 +338,10 @@ module HTTPX
       purge_after_closed
       @write_buffer.clear
       transition(:idle)
-      @parser = nil if @parser
+      return unless @parser
+
+      enqueue_pending_requests_from_parser(parser)
+      @parser = nil
     end
 
     def used?
@@ -375,6 +393,19 @@ module HTTPX
       current_session.deselect_connection(self, current_selector, @cloned)
     end
 
+    def on_connect_error(e)
+      # connect errors, exit gracefully
+      error = ConnectionError.new(e.message)
+      error.set_backtrace(e.backtrace)
+      handle_connect_error(error) if connecting?
+      force_close
+    end
+
+    def on_io_error(e)
+      on_error(e)
+      force_close(true)
+    end
+
     def on_error(error, request = nil)
       if error.is_a?(OperationTimeoutError)
 
@@ -493,7 +524,7 @@ module HTTPX
           # flush as many bytes as the sockets allow.
           #
           loop do
-            # buffer has been drainned, mark and exit the write loop.
+            # buffer has been drained, mark and exit the write loop.
             if @write_buffer.empty?
               # we only mark as drained on the first loop
               write_drained = write_drained.nil? && @inflight.positive?
@@ -578,6 +609,7 @@ module HTTPX
     end
 
     def enqueue_pending_requests_from_parser(parser)
+      parser.reset_requests # move sequential requests back to pending queue.
       parser_pending_requests = parser.pending
 
       return if parser_pending_requests.empty?
@@ -586,11 +618,14 @@ module HTTPX
       # back to the pending list before the parser is reset.
       @inflight -= parser_pending_requests.size
       @pending.unshift(*parser_pending_requests)
+
+      parser.pending.clear
     end
 
     def build_parser(protocol = @io.protocol)
       parser = parser_type(protocol).new(@write_buffer, @options)
       set_parser_callbacks(parser)
+      parser.max_concurrent_requests = @max_concurrent_requests if @max_concurrent_requests && parser.respond_to?(:max_concurrent_requests=)
       parser
     end
 
@@ -627,7 +662,6 @@ module HTTPX
       end
       parser.on(:close) do
         reset
-        disconnect
       end
       parser.on(:close_handshake) do
         consume unless @state == :closed
@@ -684,11 +718,7 @@ module HTTPX
            Errno::ENOENT,
            SocketError,
            IOError => e
-      # connect errors, exit gracefully
-      error = ConnectionError.new(e.message)
-      error.set_backtrace(e.backtrace)
-      handle_connect_error(error) if connecting?
-      force_close
+      on_connect_error(e)
     rescue TLSError, ::HTTP2::Error::ProtocolError, ::HTTP2::Error::HandshakeError => e
       # connect errors, exit gracefully
       handle_error(e)
@@ -719,10 +749,10 @@ module HTTPX
       when :inactive
         return unless @state == :open
 
+        # @type ivar @parser: HTTP1 | HTTP2
+
         # do not deactivate connection in use
         return if @inflight.positive? || @parser.waiting_for_ping?
-
-        disconnect
       when :closing
         return unless connecting? || @state == :open
 
@@ -740,8 +770,6 @@ module HTTPX
         return unless @write_buffer.empty?
 
         purge_after_closed
-        disconnect if @pending.empty?
-
       when :already_open
         nextstate = :open
         # the first check for given io readiness must still use a timeout.
@@ -758,6 +786,30 @@ module HTTPX
       end
       log(level: 3) { "#{@state} -> #{nextstate}" }
       @state = nextstate
+      # post state change
+      case nextstate
+      when :inactive
+        disconnect
+      when :closed
+        # TODO: should this raise an error instead?
+        return unless @pending.empty?
+
+        disconnect
+      end
+    end
+
+    def force_purge
+      return if @state == :closed
+
+      @state = :closed
+      @write_buffer.clear
+      begin
+        purge_after_closed
+      rescue IOError
+        # may be raised when closing the socket.
+        # due to connection reuse / fiber scheduling, it may
+        # have been reopened, to bail out in that case.
+      end
     end
 
     def close_sibling

data/lib/httpx/plugins/auth/digest.rb

@@ -8,7 +8,8 @@ module HTTPX
   module Plugins
     module Authentication
       class Digest
-        Error = Class.new(Error)
+        class Error < Error
+        end
 
         def initialize(user, password, hashed: false, **)
           @user = user

data/lib/httpx/plugins/brotli.rb

@@ -3,11 +3,33 @@
 module HTTPX
   module Plugins
     module Brotli
+      class Error < HTTPX::Error; end
+
       class Deflater < Transcoder::Deflater
+        def initialize(body)
+          @compressor = ::Brotli::Compressor.new
+          super
+        end
+
         def deflate(chunk)
-          return unless chunk
+          return @compressor.process(chunk) << @compressor.flush if chunk
+
+          @compressor.finish
+        end
+      end
+
+      class Inflater
+        def initialize(bytesize)
+          @inflater = ::Brotli::Decompressor.new
+          @bytesize = bytesize
+        end
+
+        def call(chunk)
+          buffer = @inflater.process(chunk)
+          @bytesize -= chunk.bytesize
+          raise Error, "Unexpected end of compressed stream" if @bytesize <= 0 && !@inflater.finished?
 
-          ::Brotli.deflate(chunk)
+          buffer
         end
       end
 
@@ -30,19 +52,25 @@ module HTTPX
       module_function
 
       def load_dependencies(*)
+        gem "brotli", ">= 0.8.0"
         require "brotli"
       end
 
       def self.extra_options(options)
-        options.merge(supported_compression_formats: %w[br] + options.supported_compression_formats)
+        supported_compression_formats = (%w[br] + options.supported_compression_formats).freeze
+        options.merge(
+          supported_compression_formats: supported_compression_formats,
+          headers: options.headers_class.new(options.headers.merge("accept-encoding" => supported_compression_formats))
+        )
       end
 
       def encode(body)
         Deflater.new(body)
       end
 
-      def decode(_response, **)
-        ::Brotli.method(:inflate)
+      def decode(response, bytesize: nil)
+        bytesize ||= response.headers.key?("content-length") ? response.headers["content-length"].to_i : Float::INFINITY
+        Inflater.new(bytesize)
       end
     end
     register_plugin :brotli, Brotli